{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autograd (2)：深度前馈网络——前向传播与反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 创建时间：2019-12-20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这一份文档将会回顾比较多的内容。\n",
    "\n",
    "- 第一段中，我们会使用以向量模长为学习目标的模型训练过程，来作最为简单的深度网络模型：多层感知的学习与热身。\n",
    "\n",
    "- 第二段中，我们会了解网络中的参数如何调取与浏览，并且解释多层感知模型本质上几乎就是普通的矩阵与向量运算，学习前向传播过程。\n",
    "\n",
    "- 网络参数导数是模型训练过程中非常重要的参考。第三段中，我们会了解网络参数关于损失函数的导数，学习反向传播过程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "\n",
    "from IPython.display import Image\n",
    "\n",
    "torch.set_printoptions(precision=5, sci_mode=False, linewidth=120)\n",
    "torch.Tensor.backward.__defaults__ = (None, True, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch 多层感知 (MLP) 学习向量模长的简易例子"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 小样本学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相信绝大多数脚本砖工在入门深度学习的时候，都是拿一些现实的问题，譬如 MNIST 数据集或 Iris 数据集来作训练。这可以加深砖工们学习的积极性与目的性，训练的结果也确实可以有效地泛化 (不严格地说，泛化类似于实用化)，固然不错。\n",
    "\n",
    "但这里尝试用另一种方式来作入门级说明。我们拿一种事实上不太适合深度学习的问题来举例，不管是从问题的意义上还是实用性的角度上。这个问题是用多层感知 (multi-layer perceptron) 模型，学习固定长度 (5 长度) 向量的模长。但这样一个问题非常容易用数学的方式作定义，其训练集与学习目标都是完全可重复、不需要任何先验的知识就可以构建；并且不需要脚本砖工最关心和头疼的数据集处理 (特征向量处理) 的过程。\n",
    "\n",
    "在 PyTorch 中，这样一个简单的模型 `model` 可以通过下述简单的 layer 堆叠得到："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=5, out_features=4, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=4, out_features=3, bias=True)\n",
       "  (3): ReLU()\n",
       "  (4): Linear(in_features=3, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.random.manual_seed(0)\n",
    "model = nn.Sequential(nn.Linear(5, 4), nn.ReLU(), nn.Linear(4, 3), nn.ReLU(), nn.Linear(3, 1))\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于上述每一段代码的意义，后文会作更详细的描述。下面的图片就是对上述模型的一个简单描述："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2sAAAIDCAYAAACaZYYTAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAOkbSURBVHhe7P37uxTVmf+N5x/x7/DnjxqPJLNDohHHTGZymDhP5pvPRElIfGQ0MspoxAOJoii4jUZFJERUIGwlIIpKUAiioCQEPKLGaNCQGM1T3/1au+/NslxVterUXd39fl3Xuth0V1fVqlpdfb/XfVifSYQQQgghhBBCdA6JNSGEEEIIIYToIBJrQgghhBBCCNFBJNaEEEIIIYQQooNIrAkhhBBCCCFEB5FYE0IIIYQQQogOIrEmhBBCCCGEEB1EYk0IIYQQQgghOojEmhBCCCGEEEJ0EIk1IYQQQgghhOggEmtCDJijR48mF1xwQXLo0KHeK6MLfaSv9FkIIYQQQuQjsSZKg8G9cOFC18ZBYLQJomViYiL5zGc+40TMqHPiiSe6vtJnCTYhhBBCiHwk1sYEhEBTgsAM7nERGG1iQu2EE05Ijhw50nv10+zfv3/2mue1ft+PycnJTxyf88wDcU9f2Za+CyGEEEKIbCTWxoQmxZoZ2zSEm6iGL3SKPJRdFGucc/r4RWIN/L5wDYQQQgghRBiJtTGhSbG2fft2J9hou3fv7r0qyuALlqmpqd6r2aS35/+h1s+wVBtTvnjnHGJYu3Zt6c8IIYQQQowbEmtjQpNiTdTHQklj7weCpkviBsFo54N4r3Judg3knRVCCCGECCOxNiZIrHWHKuKmS2KNwiDmTcNDVvXc/OvA30IIIYQQ4pNIrBXgG6KAoYqB6hfZoCpilpFa9/NNUVes2bmmG30pwkLe7NgU0rjuuutmDX7+5f95BTYMjHrri3029voRsslxrKiHNf7POeZVJ0z3we6jvy/+jgkLLetVA38ctT1WiuB6cx7mEatzbnYtVGxECCGEEOLTSKwV4BuifiW7UAsZqnU/3xQmcMoIBJ/0uVorK9Z8b0q6cW2yBBOvWx+yGkIsCzuHvMbxswSj/3nuoy+20y3PS+QX5SjjTfLHUZvjpIjQedQ5Nz+csp/5dkIIIYQQw4DEWgG+IWpCyzwovOcb8SHvQN3PN0Vdsca5+s3OmfMvwvroC1XzhtHMU0PLqg5o588+MPBN1GHg+5/POh9e57MIOr9AR9pTx75C+PfJ+sHn+Lztx17n3yz8/eR58tJwDPscfw8KE6m+MK5zbohj+yz3RQghhBBCHEdirQDfEKWFvDe8Zu+nqfv5pqgr1tLY+WaJIx9foNBCHqW80EC2t89mhRmaYEMohURQUYilL9hCpPsQEhb+eWaJFjtOWWGeHkfW6C/79AVsW9g1SF9j/9yy+p2HidwsoSyEEEIIMa5IrBXgG6IhoQUIiCxjte7nm8JEQkgMVcHOt6xYCwk18NccS2NCLE/g+OGFWYIuD/8cQ8T0wb/XWffRRGlZYeLvO6sheqr0PQbfA5buf0y/86gqYIUQQgghRh2JtQJiDNG8bep+vim6ItayyNvGXi86Vux2IYrOMaYPMdg+yp4jnizGRrqxH7u31mIKtZQlb/xwHnZs/i5LPzzLQgghhBDDiKyjAmIM0bxt6n6+KUZBrMW2rHPC+4YwwIMT+py1EDF9iMH2EXPdykAYpO07y4NbFT+8MyQE647fpq6tEEIIIcSoIeuogBhDNG+bup9vinEXa77gKGohmhIUto+Y61YWy/1q6h4DHj0/p4zxmW6+UORvez3WwyfPmhBCCCFEGFlHBWB0miHJ3yHytqn7+aYYBbFWVeD41xevWiivq+gcY/oQQ5vFNMxj2KRY869d2RZ7v5oem0IIIYQQo4LEWgExQipvm7qfD4ERjGHLv7EVAJs2iO18YwzyGKGTt01dEeLvO+t6FZ1jTB9isPvQdDENvwBIbBgkY63I++WPzbKNaxZD1aIrQgghhBCjjsRaATFCKm+bup9P44ec0WIN82EWa/57WZUYjZAY8z8fep9r7i9yHSKmDzEUnUsV2I+fhxezuLS/fdE1LaLM+E3ji0zGthBCCCGEOI7EWgExhmjeNnU/n8ZEl99i8D06HCOvpcGgTm9jx0Ys+q+HhEKM0MnbBjFi4YM0+uLnRhHWyOfpW0iM+gKXbdiezyFSYq9nU2LNX2KgjEgK3QP6wZIH/rWJ2Sefte1pfL4O/v74uwycr322yMsnhBBCCDFu1LM8x4AYQzRvm7qfT+OvR0YLiZMQIVGS1dL4QqWohc6nrlgDRI7v/cpqoVC6tOcp1Pic/R2iKbEG1g/OKZaie4DgihV/XI/0tYzxxmVRZvymsfsSO46FEEIIIcaJ+pbniBNjiOZtU/fzaTC0TXhh6MYa2cMu1oC+4yVL9wXhgYcvT6zwWYSuL1L4m9fMo2Ovh2hSrPnepKL7bYTuAQKNa8E1oX9l8D18tNjzCFFm/Pr4n4sVmkIIIYQQ40R9y1MIUZoq3rWmMaFEGwTmVRvkNRBCCCGE6DISa0IMAPLNTCjhNes3vmdtEFUYfU9hnRBMIYQQQohRRmJNiAFB6KYJljphiGUgXJKQQytKwr/9Luzhhz8OQqgKIYQQQgwLEmtCDAiEk4UCIpra9jClc/36ccw0HM+EYii/UQghhBBCHEdiTYgB4gs28tjaxMRa1aIkTeDn6g3i+EIIIYQQw4TEmhADBtGCgBqH3C36SF8l1IQQQgghipFYE0IIIYQQQogOIrEmhBBCCCGEEB1EYk0IIYQQQgghOojEmhBCCCGEEEJ0EIk1IYQQQgghhOggEmtCCCGEEEII0UEk1oQQQgghhBCig0isCSGEEEIIIUQHkVgTouOUWTR7+/btyXXXXdf7nxBCCCGEGGYk1oToMAi1iYmJ5DOf+YwTbHkcOXLEbUeTYBNCCCGEGH4k1sTA2b9//6zISLcTTzzRCQ+2aRP/HMoeCxHF54rEVJVjmFA74YQTnBgrAs+aHWNycrL3qhBCCCGEGEYk1sTA8UVMXpuamup9onm6KNYQW7Z9TAiksXbt2tnP7d69u/eqEEIIIYQYNiTWxMDxRQyCjP/TEBp41ew9WozIqYJ/DmWP0YZY87etIlLtnPDIEUophBBCCCGGD4k1MXCKRAyizd5vKxerjJBK04ZYI/wzZp9ZKH9NCCGEEGL4kVgTAydGxNj7VcVLEWWEVJqmxZqfd1b2XHx8r2RMvpsQQgghhOgWEmsjQNqwx9i3whQ0/ua1LExskOsE7GfhwoWzn8fLw3tthdPFiBh7n/PKw86d8D/7DP3L6z/EnEMWTYu1ul41w/euqdiIEEIIIcTwIbE2AphBTrigL9LSzcRYGl+spXPE/FZXPGRRJGL89/NEl1+QI9S4NlmCM1ZIhWhSrFFIxLYpEpgxmOhGvAohhBBCiOFCYm0EMOPevEn8i6GPIOBf38sUEismNvztEG58Pi0A26gumCVi8AxxHnZenEcWbGf7QLRZ2J/tw97LElRZ5xBDk2LNP9cmPJncf9tfmYqSQgghhBBi8EisjQBmjNNC3iPfYA+JBBMbNIRR2qj3w+kQE03ji5isxjlmiZeYcD8qKto2oWsQI6SyaFKs2b7yhGkZfE9dm0sfCCGEEEKI5pFYGwHMGM8K8+M12yYktkwghISaYd61IkFSBV/EhFpROKAvxPK8UeahCwm6rog1y1crys0rgx1TeWtCCCGEEMOFxNoIYMZ4ntcrb5sYsRErSKoQEjGIRhOIiKw8ERZ7bnnbxQipLGKPH3MMe79JD2ZTBUuEEEIIIUR/kVgbAWIM/LxtYsRGrCCpQpaIQaCZNyyvb3ZusS3UhxghlUXstYk5hr3fpFiLPT8hhBBCCNEtJNZGgBgDP2+bGGO+TYM/T8Rwvrye512zc4ttoT7ECKksYq+Nv7h3P8WaPGtCCCGEEMOJxNoIEGPg520TIzZiBUkV8oRSUb4dNHFuTYg1RFEeJjxpWcLTPIlt5Kw1KQCFEEIIIUT7SKyNADHGeN42MWKnjCAi3wyxQYsRPkVCydZ+yxJDtr5aUW5bHnXEmi/CbMmAELbmWZ6os+vcVDVIv1KmqkEKIYQQQgwXEmsjgBnjXRFr5h2ylidgoEgo+eXnQ5Uh/c8j7PLIEnN1xJof3ph1fH//effJF35VhaeP1lkTQgghhBheJNZGADPGuyDWfFFiLe+8IEYo2fGzvFL2vm3DMdmXNf5vnq0Q/jnggfI/m24h8WmVK2mcCyLJtjfPH63I+1ckTMtS5JUUQgghhBDdRWJtBDDjvgtiDSFix7KGYMmD94u29T1EIRHDce0c8xpiKYR/DkUtdA0RcL5gCzWOHePdQlixfd1QSP9e5I0NIYQQQgjRTSTWRoAYgzxvmybFGiCmECa0mIWYY8QasD+2yfMScWw8aLYtjb95DY9ZllerrlgD9s0x0qIR0cVnso6dxhemedejCI5p+ykKRRVCCCGEEN1DYk2IDlLXu4Y4M8EaI5iFEEIIIUT3kFgTooP4RUvyPKZZmHcP0Rfr0RNCCCGEEN1CYk2IjmLFQWhlwiH98Mc6YZRCCCGEEGKwSKwJ0VHwiFnREkIaY4qT+PluVTxyQgghhBCiO0isCdFhfMFWVH6fPDUTakXrzQkhhBBCiO4jsSZEx0GwkYMW61mTUBNCCCGEGA0k1oQQQgghhBCig0isCSGEEEIIIUQHkVgTQgghhBBCiA4isSaEEEIIIYQQHURiTQghhBBCCCE6iMSaEEIIIYQQQnQQiTUhhBBCCCGE6CASa0IIIYQQQgjRQSTWhBBCNMo4Lc5OP+mvEEII0QYSa0KIsWft2rXJBRdc4P4V9UC4fOYzn3Ft//79vVdHE7+vEmxCCCHaQGJNCDF0IKrMSK6Lb3DTRl1gtMnu3btnr+Pk5GTv1Xj4TL/uxZEjR5KpqSkn0k844YTZY05MTLjz4P0Y8KzZZ+m/EEII0SQSa0KIoaNJsYbBbvuiyUNSDcSNiZ6FCxf2Xo3n0KFDn7gPtDbFGiItfbx0ix0Lti/6HyvyhBBCiBgk1oQQQ0eTYu3o0aOzxjb/imrYNTzxxBPdNS2LL3js3rYt1vCiMZY4Dg3PmO8po8WIL/pr560xJIQQokkk1oQQQ0eTYk3Uxw8lrSKwfO9m3X3Fkico/fPh7xj885Z3VgghRFPI0hFCDB0Sa90Cbxr3oopXyfdKmZfL7m2bYi0PzsnOgXOKxa4D/wohhBBNIEtHiDEjbQjjBSAczF7n7zzPgIWrmRHLfshRss9jqPJelVC4WOqKNf/z6VaELyaAfrI/M9RpXI8YoUGIHWF3/mctNK/o+pHjRSEMux/W2Bf7zAvfS/cBuOf+vuw+FuF7lGL6nMbGDscD/9yq7K8JuHZ2DrGeNfCvhYqNCCGEaAKJNSHGDN+Y9EVaumUZ6r5YS+f3+K2KlyUWjm3HqYL/+XQrwhcTCCY/xyrd8sQGhn3eZ3mP/YfwzyGvZQkG//NskxZ8fuMe55EWW2Xwz8OuVei1fmPjg3tQtmCI3dMqRVaEEEKINBJrQowZZgibUcm/CAcM47SACHl3zLD3t8O45fNpAdiWd6GuWMMA53yt+aKzCLa3be0a0Gf6ynv+ufF6CH8fXE8TZVxv/x7wb+ge2OcRBHbtaZyD3xc+H8I/vh0LsYUXyfbj38cs0Qi2TZVS/RyTz/qC0D83/u4njAv//nEvymLXP+vaCyGEEGWQWBNizDBDlIZBnhYDGKj2fshY9r0wGKRpQ94PIcPwbYO6Yi1Nmf35YoIW8jwViT8TKVlijmtqnw9dQ+5ZSMQZfn9C9zDUh/T+Yu6jv5+ywtzOMS1I/X2Gzr1p7Fh+C43rWPziJFX3IYQQQhgSa0KMGWZIhoQa8JptEzLSfc9aljFqXhm2bQNfjDRBmf35YiIk1ADhYtukBYcvxPIEjoUXZgm6PIoET0wfwLbJEmu+sC8jTHwhmPZeFZ1709ix0o3xXSZfzfDPv6yAFUIIIdJIrAkxZpghmWWAQ942JtbyhFjMNnXoiljLEhN528Qeq04fi86x6P1Yqp5j3vho6tyqwEQFAsvOj1Y2vLMfnmUhhBDjQzOWjhBiaIgxJPO2iRFiMdvUoY6QCVFmfzFiIm8b/1ixLYTlV/nCItRC5xjThxiq3AffG0cf0jR1bnUxz2bWeeZhn+P6CCGEEHWQWBNizIgxJPO2kVgbvFgj5JAwvdC26RY6x5g+xFD2PuC5svNGDHHsdPNzvqzgCa2sYKoLx/TPIxZ51oQQQjSJxJoQY0aMIZm3jcRac2KtCr7g4V88Vencw6JzLHo/lrLFNPzjlm1547UN/HMtI9aaurZCCCEESKwJMWaYIZln/OZt07RYsxwhCl3Eek/qCp40ZfYXY4znbZNXfCQGf99ZBSzyjg9F78cScy4+/vZlW5FYY+zU6UuaqsVTqn5OCCGECCGxJsSYYYZkF8Qaxqwdixa7sPIwizXfM5ZVkdMnz2sWEkiIFrv+oeND3vmVxfZTZZ21EFXOzRfAMde0CK6hLa8QOyYNW7aBeyyEEELURWJNiDHDjNouiDVfJFmLMdD9z7F9XgsZ7ultzMAO7S9PLPF3iKJt/PPHqEfoIDjYlkbYHefEe+nP+wLXwiDZhs/7/bAWOj6v5b1fBivEUVbUZFHl3HxxSisKW+Sesu9041radWc//FvWO2Yij/0IIYQQdZFY6yN7Dh5J1m57Pln+0G+Sq+7emnz3xoeTry6+PznzB3e4dv4V97nXeI9t7t+6133m43/8o7cHIepjBm0XxJrvEaFhHMd4RUIiL6thhKcJbZfV0p/n/1nvGTHbINBsm7wWEgshUeY3v5Jh6Pgx5xeLH/YX8vSVpcq5+blzNPqfh3+MrIboKivU/PFc97oKIYQQILHWIh9+9HHy9L5XkmtWbUvO+dE9yf/57m2V2sQld7l9bNtzyO1TiDqYMdkFsQYmWjCOMfxjGAWxBryH8DJvjDWuG33My+FDoNji4zSELvuy49k9CB0/9vxisfOPud9FVD03X6AWnYd/DL/RD65h7DhMY9ec/QghhBBNILHWAgde/WNy5V1bnLfMF12fXfDzZM4V65OJxVPJF67Zknzp+ieTL9/4THLeLXtc429e4z22YdtTf/hJkcc+L5t81B1DCCG6QNPetSr4Aj5vIqItfK/aoK6BEEKI0UNirUFeffvPLoTxlItW9sTViuS0i1c54YUQ++dbn6/Uzr3p2WTiqkeS0y9Z7fZpwm3hyqnk8Jvv9o4uhBCDw7x8saGsTeN71sqGL9aF/lqeW1EIphBCCFEGibUGeOfoseTa1Y/PirSTLlyRfO6Kjcm8m3cHxVedxj4/f+XG5OT5t7tjcUwEIkJRCCEGBQLJBEsT4ZCxcFxfqDVVlbIMFv44KKEqhBBidJFYq8mW3QeTMxZM9rxdK5I5lz+YnLuseZGWbhxjzqKH3TE59qnzVyYbdrzUOyshhOg/fjhk29UQQ3lnbR8zBMe04zeR/yeEEEL4SKzVYHLjMz2RdltyxsI1tUIdqzaOeealv5w9j5+ufVLVI4UQA8MXbFULdcRgYs2KggxCKPWrr0IIIcYXibUKHPvb312+2IxAWuFy0kJCqp9t4urN7lw4p/nL1ifvH/uwd7ZCCNFfEC6D8HINAvopoSaEEKItJNZK8ta7HyRfu3qNE0Unz59MvnT99qB4GkQ7e+lT0+c0U4GSNdtUfEQIIYQQQojhRWKtBHjUTKhRUp8qjSHRNMhGLhsVKE2wvffBX3tnL4QQQgghhBgmJNYiIQ/s4ls39YTa3W5dtJBY6kLj3E67eGZ9tu/e+LBy2IQQQgghhBhCJNYiWbbuaSd+CDPsokct3fCwWUjkNau29XohhBBCCCGEGBYk1iJ46Mn9TvRQwIO8sJA46mI7e+kOl1fHud/9yO5eb4QQQgghhBDDgMRaAW+8835yam8B6i9csyUoirrc5i7Z6s6dxbNVcEQIIYQQQojhQWKtgCvv2uLEzlmXPRAUQ8PQ5ix6yPWBnDshhBBCCCHEcCCxlsP+l992IuekC1cORZ5aVpt3M/lrM97BXQde7/VOCCGEEEII0WUk1nKgkiIC53NXbAiKoGFq/7R4ppLlN69Z2+udEEIMhthFsw8dOpRccMEFydGjR3uvCCGEEOOFxFoG2/YccuKGiorzbu5umf7Ydt7y55JTvn+n69OGHS/1eimEEP0FofaZz3zGtf379/deDXPiiSe67SYmJiTYhBBCjCUSaxl8Z+mDTthMXPVIUPwMY6NAirxrQrQP3iATJOnGe1NTU62LDzsH/i3D2rVrZ8+1CNuOz8Swe/fu2c9MTk72Xs0Gz9oJJ5zgtkewCSGEEOOGxFqAd44ec6LmpAtXdHrx67IN79pJF83krr369p97vRVCNE2eWLOG+Dhy5EjvE83TNbFGX014LVy4sPdqMXjf7DgxAk8IIYQYJSTWAti6aqdfsjooeoa5UdWSvmndNSHaw4QSggyxYQ2PmoX20coKqTJ0TazZ+dD/sl5F/5yKQieFEEKIUUJiLcD8ZeudoJm4enNQ8Axzs1BIwjyFEO1QJJQQcSY+2vKudUmslclTy8JELv8KIYQQ44LEWopjf/u7W0AaQXPuTbuCgmeYG2GdLEVA/95694Ner4UQTVIklPrhKeqSWDOhVfZcfHzBx99CCCHEOCCxlmLL7oNOyJx28aqg2BmFRngnfVz3xL5er4XoDiYWzLDH80SZd8t34l/+n+WRqvv5JigSSr4goohGFnbufugkXjk+XxRKWHQOWfjnVoRtx2eyaMKrZth1ULERIYQQ44LEWoo7N+1yQmbOooeDQmcU2uev3Oj6uGL9zl6vhegOvtjyDf10Q3SFBEvdzzdBkVCy9/NC+jh3E5ihxnt5Qq/oHLJoWqxRTIRtmghfJOfPjpnXdyGEEGJUkFhLccOa7U7ITCyeCgqdUWhfuHqz6+NVd2/t9VqI7mBiwRcqGPx4ZWhm/NNC1QHrfr4JsoQSpevtPRr/D8F52jZsb8IEcemLOP7NEpxZ51BE02LNtmniWuNptP0h3IQQQohRR2ItxcKVU07IzF2yNSh0RqHNXbLN9XHBLRt7vRaiO/higYY4SZOXA1X3801gQimrIbJC52UUhfsh3mxfWUKpC2LNF51ZwrQsJlQR3UIIIcSoI7GW4tvXr3NC5uylTwWFzii0s5fucH3U4tiii/hiIUvQ4KWxbdLU/XwT5Ik1BFhe+KUvxPIEjnkIswRdF8Qa19+2aSps0fqlvDUhhBDjgMRainN+dI8TMufe9GxQ6IxCo8olfaSvQnSNGLGQt03dzzdBWighzvx8qyxxA7HnVrRdF8RamX3FQsGVpvcphBBCdBX92qVAxNBCImeUmhOky3YlP3rgD8mBN4/1ei/E4Ikx8PO2qfv5JsgSSnbcvFwz/9xiW4hRFWtt7FMIIYToKvq1SzFOnrV/vuW3yVduez757r0HkgX3/y7Z9Pw7yV8+/Lh3JYQYDDHGeN42dT/fBFlCCYFmOVecQwj/3GJbiLbFGn2x7WL60hTyrAkhhBgn9GuXYpxy1v7t+qnk/FtfSL676kDy7OH3k1u2vpZ88479yU83v5K88LoWzBaDIcbAz9um7uebIE8oWb5clnetqXNrQqzlrUUXUzykjVL7VfslhBBCDCPtWCpDzDhUg/zidY+7Ps5ftj75xrQ4+87dLyaXP/iH5ON//H/Os4aHDU/b91b/Lnlozx+To3+Vt030j7piq+7n0yCo8OYgDvIqOPrkCQq//Hxofwgfex9BVJWqosYXYXn99YVYlqhroxqkVcpUNUghhBDjgMRaimtXzwiZUV5nbcJbZ+3up99I/nXlvuS/HziY3PTrV3tXYQZy2VY+/npywZ37kxseeSX57Svv994Roj26Jtb8ddloMYKtSCjZPkMLRfuhkkWVIyHr/apijf3xubzj++dYtH/bl9ZZE0IIIcojsZbizk0z+VxzFj0cFDqj0D5/5UbXxxXrdyZH/vxh8o3JfcnVGw47T9rqnW/2rsRx/vr3fySPvPCn5JK1B5P/uvelZN3ut5P3jn3Ue1eIZumaWLPtrMWInyKhVOS98s8PUYTQwTPF52gIFbx9vMf/Q9g5ILjsc1ktjb+0AYKS49m2/G1CjRb6vE+eMC0L18qOmxeiKYQQQowKEmspHn32907InHbxqqDQGYV2xsI1ro9rtz3v+vzfvzyY/Psd+5NnDx9N/p+7Xky2vviuez3EwbePJZNPvOG8bdduetl9hvBJIZqia2INsWPb0vhsEUViDSycL0vE+IIpr2Xlgtk5xLQQaY9iqMV4GX2BVTcU0u5F3nUVQgghRoliS2XMeO+DvyanXLTSiZl5N+8Oip1hbufdsic56cKZ/r317kwREec1mxZs//PQISfGKDJSVGDk7x//f07UXfrAweT/d/dLyZpn3kre+eDvvXeFqE7XxBpiyIQVAqYoLBFixJqf85UlevBa4UGz41tjv/Qhz7tUV6wB50WffU8a58I5lfFs2fnXEVlcCzuHGJEohBBCjALFlsoYQuENxAy5XSHBM8xt7pLHXN++ec1x7wBFRQiF/M9p0YVI23noqPOcvfbu33pb5PPKn/6W3PnkG84rd9WGw8nTB/8sb5sQYpYmvGvmVeNfIYQQYlyQWAtAeCCChnDBkOAZ5nbWZQ+4vpGb50O5/uVbX3feNaAK5IWrDpTKTcPbtv13f3b7QLjdu+PN5K2j8rYJIY6LLbx0Md5JH98T2tQSAEIIIcQwILEWgPBABA3hgoQNhkTPcLa9ycnzb3d9O/zmJ/PSyD370bo/OIHG30BuGmGOFBgpC165u5864kTb4vWHnIhDzAkhxhNEloVTlgmH9MMfEW1CCCHEOCGxlsF3lj7oRM0ohUJaCOT5V9zX6+VxCFtEWD342z8mF//i97OvXfOrl13Z/qqwD8IiCY8ktBIBFxteKYQYLfxwSPLeiqgq8IQQQohRQWItg007Dzhhc8r37kjOW/5cUPwMV9ubfHbBXa5P657Y1+vlJ0FIUbofsUbeGuBV4/+ENNaFkEj2Q0ESQiUpUCJvmxDjhS/YigqFWGESQijLhk4KIYQQo4DEWg4XXPtLJ25YlywsgIanTVz1iOvLVxffn3z8j3BYI4VCWEcNocaaa1YkhLw1vG6/3p9d0r8M7JdQS7x2eNsIt+TYQojxAJEW61nDoyahJoQQYlyRWMth14HXncA56aLbk3OXDW8Z/3k375nNVdv+/OFe78LgRaMiJP/6660hpijp/9tX3u+90gyU+6fsP9428uNYRqBKjpwQQgghhBCjhsRaAQtXTjmRM+fyB4NCaBjanCtmliL47o0P93qVzYbn3klu2fpa8tyrH7hiI+ZdA4Qagq0tLxj7Jz+OY6x8/HW35psQQgghhBDjisRaAa++/efZRbLnLtkaFENdbl+87onpc1/hzv/Aq3/s9Sqbo3/92IklcsnIK0uHPuL5wgtWpqR/Wdj3ut1vu5DMS9YeTDY9/45bC070jz0Hj7glLJY/9Jvkqru3OqFPCO2ZP7jDNYrU8Brvsc39W/e6z2SF2AohhMhmnEJ+Fd4sRDkk1iJYtXmPEzsnz59Mzl66IyiKuti+fONOd86c++TGZ3q9KebaTS8njx94z4VDkquWLgJCIRJEVD/CFTkH1oBDQOLxO/CmvG1t8OFHHydP73sluWbVtuScH93jxkyVNnHJXW4f2/YccvsUQowmLG6OwU3uoYzuevhVT8dheQoVDhKiHBJrkWCAYoxSHXIY8tfm3bx7tvrjop9t7vUiDgqMsDYa8C+hkWkIV0TU+WGSbYLHj4W6KXyy4P7fuXPiNVEPvK1X3rXFect80fXZBT934bMTi6eSL1yzJfnS9U9Oi/9n3LqDNP7mNd5jG7Y99YefFHns87LJR6M8ukKI9rEqnHUFwZEjR2b31cT+xhmupQk1xEseCBuK8yxcuHBW8ND4G9HMmoT9hHPxzyMWX5wW9VkIIbEWDeFdhH1hhJ528T0dXyx7b3L6Jfe5c6Wi5bG//b3XizjwpFGlkeIfh/7416B3Da8aBUGo5Nhv9r/xF+dlw9uG1w3vmygH4b2EMFqIL6Gyp128ygkvhFh4XBW3c2961lUePf2S1W6fJtzI/UwvxC6E6C9mVNcVVxjbti9aTGVPEQaxwjVE9BR5mbhv/nUPtbbvBefIefgizVoZ/MXuJycne68KIUJIrJXg/WMfurydGcG2qpMeNjxqM4bybS6c7Z2j1cIGEWFrn33b/Y0HjYqNacgtI6+MnLJBQB4bx8bThseNPLc2c+lGAcbDtasfnxVpJ124IvncFRvduAmNpzqNfbLshVUi5ZgIRISiEKL/mHHchCcMA5t9YbTjHRLl8cUXArgIE0lce8JQETw0/m+eKlrR+oVV4fj+cQiDNbFJK4vf/357BYUYJiTWSoJ3gOIKGJ8nz78jOXvpU0FDdRDtnJ/sSE75/p2zQq2OJ4NKjFSDBLxreNpCRT5ee/dv7j3WTRsk5LJRQRJvGyGaTS8xMAps2X0wOWPBTA4jXi8qnPZjwoFjzFmEV3rG03bq/JXJhh0v9c5KCNEvzDBuQqyJevihpLH3I8/z5ns7CZNsAxNmCDbEIiDY7LhVMA8d/wohwkisVeC9D/6azF82Uw4fA3Ti6s1BI7WfjdwhKyby7evXVfao+eCxsoIehB2GvGtAGCIiCVE3aAjPpGIlBVDw+uEdJJxz3KHAzMx4vS05Y+GaWqGOVRvHPPPSmYXmaT9d+6SqRwrRR8yollgbPIQrci8QPk0V2TAxhYBqAzx2ePH8860r1tinfb4tj6AQw47EWkUwMm9Ys33W8MQIPfemXUEjtc2G1+Ksyx+cPQ/CzJoygCnogbcKjvz5QyfIssIMWUCb3LYuCSO8g4Rz4vm75lcvO+9fvwqidAXyFW2tQDexsHgqOI762ZjcMC8bkx6EFwvRZSxcy4xgvCIY2xYSxr/8Pysc0M/PgVDeD96QtkPB7FhVxJrfh3SLOW8z6u3YfIY+2z64FrwXI1ww6n2RwPWPuX7sm8+mC3TQ2F+RWEj3wcZB+j4WhTRW8arFYOfBOfQL/z5Uxc5bxUaECCOxVpOHntw/m/9z8vS/VMWbd3P7xUc4BvlAJ110PB+Ita6aBOGF0LHiInjX7nwyu6DI6p1vOm9cP0r6l4HzR0xSEIU14vAQvnV09L1tb737QfK1q9fMjM35k8mXrt8eHEuDaIQPE0bMuRFWrOIjostgUGNMmkFvxmm6ZXlJfKHjV8ILtRjhUxU7RhWB4Pch3WLO2Rc65lUKNbbLgmvri4NQY99ZFH2WlicY7PP8yzjIuo+8nifYpqamZrfNEvhl8e+PhSj2A/+aVsW/HjG5e0KMGxJrDUDBBMrjY3g64TRtGP/T4k2tVIw8b/lzrtqeGbo0PCdtlUe/asPhZPvvZgpCIN7yvGtAdUY+01UP1it/+psTnIhQzvPpg38eSW8bHjUTapTUp0pjaDwNsuEVplAP54hgI7xYiC5iYs03zs2TQ+Nvez1U2c43pG0fiAIrEmH7t9fbwo5RRawhlKy/NN/A5v9FmFHvX0POg89yHSyEj5YlNvx9cHwTxhj4/j3I6h+fx4vDPUJs+X3xvWP8P0SoD4jD0H3M827Zfpq41+YttHPKE6ttYH2hVcX3NGZdeyHGGYm1BkEwHQ85m6m0R34QYV91qu3xWXLSzrz0F7OeNBpLCex/eaZiY1sg1AghNBA6FhoZAuFz+YN/yN2mC+Bto2//89AhF7557443XbGUUYAw2Itv3eTGyKk/vLvTy0xwbiyFYeNZOWyii/hGOC0ULmfGPsZrGgx5//Mhg9r3NrWF7b+KWEvj94m/i/CNeoRF2oNSFBroezSzxJwJNvYf8nDmebHY3gRP6B5Cug+hfsfcR3s/JOxj8M/Db6Fx2Tb+udTBrn2eyBViXJFYa4FdB15PFtyy0VvDaqbhRaBMOuJt7pJtLhQMjwfeMhp/8xrvIc7Ydma9tOPrVbFP9s0x+oGtuWbeNP79+uQ+l8OWBYtVU0qfnLdhAJF291NHnGhDvCHi0uvKDRPL1j3txgre1y561NIND5t5ill8Xoiu4Yu1LIMYwzvLaPWFTZbnAwFi28SInyrY/gcp1kJCzTDvWkgsmRDL80axXzunLEGXh51j6PgQ0wd/rIRowotk5xFq6QIgbeOfSx1sP216loUYViTWWoRQtE07D7gQyTN/cDxssWzjs3js2NcgQsVufey1TwgvctPIX8vDSvoTZjgs4BXkfAmP5NwRcIRNDhPkUM6MmxWdWlaiqJ29dIfLq+Pc736kf/kWQsRQZIBD3jYxwqas+KmC7X+QYo1/s8jbxo5VdO6x24UoOsei92No4z6zH9+j10/BY9eEVod+eJaFGFb0regTH370cbLzxddcCXWqSF42+agL+5q36F4nxmj8zWu8x8LFbLv9+cPus4Nk/xt/cYVDDNZbQ8wUhQ3yObaz8v/DBAVICI2kIAmFSShQ0nVv2xvvvJ+c2luAGs9sSBR1uc1dstWdO95jFRwRXUJi7dOUPd8YoZO3jR0rtmX1EY8bXjo/Ry3dss4xpg9FtHmf8dTZvvsVEmnXhFaHmO+YEOOKvhUiChbI9tdRo6LitZuO57JlQUgh4YXDWn0Rbxsl/8nbQ3iyFEAX1pMLceVdW5zYOeuyB4JiaBjanEUPuT6QcydEV5BY+zRlzzdG6ORtY8eKbaE++t6bvJZ1jjF9KKLt+2z77lful10TWh3kWRMiG30rRBQsLu2X7TfvWoxw4bPksHWtpH9ZqIaJSMXbxqLbLL7dlT5RaAaRc9KFK4ciTy2rUUzn5J53sF95mUIUIbH2acqeb4zQydvGjlX13P37g5AJ5ZwVnWPR+zH4eXVtVD60fQ+bWGvi2goxqkisiShszTW/zP2G596J8q4BOW6L1x/6xOeHFfrw21feT2545BW3lAGVLwcd6kn4LALnc1dsCIqgYWose0FfvnlNfYNSiCbomlijgASeCAzbMuFutv9hFGt5xUdisH3n5XPlHR+K3o+FfdCqVoPMomzxEsYR965OQRK7JrQ6WFhqv0SmEMOExJqIhkqJOw8df6iTw0WI4wuvf9B7JRsEDmKtqDDJsEF1zHW7307+696XXF7fpuffcV7HfrJtzyEnbqio2I8F2dtuVEY95ft3uj5t2PFSr5dCDI6uiTUMWtuWFivYbPthFGv+9S3qb0h82L6zxBriJrZ0f9b7sRSdSxXosz8u8pYpAN63/uZVtyzC+kKrShMVMoUYZSTWRDQU2Uh70hAniLgYCBkkHJKwyFHkuVc/cIuC421DlFJgpR98Z+mDTtiwWHpI/Axjo0CKvGuiK3RNrNl21mLFg22PV45jZLWQ4c5r/jZ+MQv+9t8LCYUYoZO3DWLExIVt4x+XwiHcAwRQ6PO+kOFv+xz7SBcbyTrHvPMrg3/tikSVj52z3+g3Hjq/DzHi3R+vtBiPFueaPr55PGnp92I9dpyv7aPM9RBiXJBYE9EgtgiFZB01A48ZxUdivGtAoRG8cRQeGVW4Pix1gDC19eb8a9Yk7xw95kQNC7B3efHrsg3vmi0A/+rboztWxHDQNbHmG8g0jh2D/5m8FhIjJlRiWuh8YoRO0TYIxrwqjtZCwoPP+mIv3Xiv6PhF78fie5LKhELaZ7IafYj1snI90p8tIi3wilrRODZsPNe9rkKMKhJrohR4jPCm+eBxo+BGLOR3Ifr65XkaJIhYrhneNrxusaI2FltX7fRLVgdFzzA3qlrSN627JgZN18SaL1oQJrEeDNt/UQsZzSZUYlpbYg3oK56p9PlwPfAY5okVrhvb+KINocD+2K/dg6zjx5xfLJwH++Jc6tw/Ps8YsD6Uwffw0YpoQ6z54z5WaAoxbkisiVIQ6pcWZuZd8/PZimBbqioWrdU2KpDHhsglr438NvLcyHery/xl652gmbh6c1DwDHOzUEjCPIUQYpSo6l1rEl8oNSFAq2BeNf4VQoSRWBOlQWS98qdPiqynD/45ufgXv+/9Lw4LFWwrRLCr4Fm89bEZbxsVJaksWaVK5rG//d0tII2gOfemXUHBM8yNsE6WIqB/b73brEdSCCEGDSLNxFKMF6ppfM8af/cb31OH11MIEUZiTZRm9c43k3t3vNn733EQa2Vz0Sh7f/mDf6gkVoYdcgBZqw1PJQKYNdxYIiGWLbsPOiFz2sWrgmJnFBrhnfRx3RP7er0WQojRgLBF8yyVCYesC149XygNwqvle/VCYbNCiONIrInSHPnzh05cpAUWoY2EQ5YRXmx7za9eTm769au9V8aTg28fc8KVXD6ux7OHjxZexzs37XJCZs6ih4NCZxTa56/c6Pq4Yv3OXq+FEGJ08AufIJraFmwmkKz145hp/D4PKvxSiGFCYk1UAm8Y4XtpeJ2CI2XAw4RXDs/SuMPadVy/Sx846Kpm4sGkgmaIG9Zsd0JmYvFUUOiMQvvC1ZtdH6+6e2uv10IIMVr44oXCI23CMawoyaAKelhxnEEIRSGGEYk1UYlf73/XVTdMQ7XDst41oNgG4qSs0BtlyAucfOIN5227asNhlxfoX9eFK6eckJm7ZGtQ6IxCm7tkm+vjgls29nothBCjB4INL9M4iJdx6qsQTSCxJipBdUMKZPBvGhbJ3vDcJ8v7x3Doj391+2y6vP2wg7eNXECuK8Lt7qeOuCqa375+nRMyZy99Kih0RqGdvXSH66MWxxZCCCHEOCKxJiqDZw0PWxrWT8NLhsgoC6GVCJJxKelfFq4LYo1rdN5PdyRnXvrLkawEaY2+IdbO+dE9vSsghBBCCDE+SKyJyiCsyFELce2mlyt514AKiYRSNrEO2ajy4Uf/SCb+d8qVtw+JnFFqTqz9ZEfyPw8ecsseCCGEEEKMCxJrojLkT1EVkuqQaSykMRQmGQPeI4QgxUfEcagaeeeTM3ls59/8THLe8uen296gyBmFZp61eT99KvmXFS8k3733gFtYnAXGq44tIYQQQohhQWJN1AJRxbprIVjwuWqFR4Qg3jn2Me6w9hoLiCNS/uvemfXYXnj9L8lXlu9xQu2cn+4ICp1RaJaz9o0l65J/u31arK06kDx7+P3klq0zi4oTiqscRyGEEEKMKhJrohZULMS7FgKPGx6gqh4QvGqUsA8twD3qWFERqkAiSm597LVZUXL0rx8n31v9OyfWzr3p2ZGuBvnF6x53Ym3+svVOpBIea4uoM67wsCFiuR4IWq6NEEIIIcSo0Bmxxor6U1NTbiV7GqVdxXBwydqDmd4NPCB1xBZ5a3iTQoVMRhGKsyDMEGiL1x9KHj/w3icKtfA3YuWGR15O/uW255I5i9aP9DprE946a4gzrst/Twv49CLq5LLZouJ4Y0NrAAohhBBCDBt9EWsIMRZgRISF8BeE9Buf0Toc3QfvBqIsBN61r0/uq1UshAqIGOnPvTqa4W4sem1eIzxE63a/7UIf0+BNQojQfrjm4LRYez45ef4d04Lt4aDQGYX2+Ss3OrG2Yv1Odw3WPvt28uONh911CoXf4o2lQA0TCIh8rqUK1QghhBBiWGldrCHUTIixCGIa/33bhlXt7f+Tk5O9LUVXIfQMMZVVDISFnSmKUQeEGscg7HIU4FqxALitncY1onhIHlxDvGpPHzw6LUQOJD+8b58TMqddvCoodEahnbFwjevj2m3Pu2uAZ5FlIZ49fLRwEXWupy0qTv4jn/EXFRdCCCGE6Dqti7XrrrvOiS4E2P79+3uvHgcxxvsnnniiE27G9u3bZwWb/7roJhjDWYYzng2EFh6kOrB/8uOG2VOC6CSED28j12znoaNR69GxDALepHf/8pH795K1v0+27v9jcspFK52YmXfz7qDYGebGsgQnXTjTv7fePe5V5VogchFjjKuiAiNcX8YO+Y+MH7yYIc+lEEIIIUTXaF2smdcsKwcNkcb7iLM0JvTIZRPdBtFBjlUWeIWyQiXLQOgbIW7DVNKfME7y9hAKnDtio0whjKcP/tl9FrGL6LhsWnTgLcJLROENxAy5XSHBM8xt7pLHXN++ec0nw6cRX1wPRBrjjmsRu4g6nlnGIl45irdwbeVtE0IIIURXaVWsIdAQWwiyEPY+LZSbZt41ctdEt8HgxWjO8lggTng/tCZbWSjXjleqy0Y2/SWXD28OwoAlDqqEcFJwhOuGFwmRQl7bym2vz4aVEh6IoCFcMCR4hrmdddkDrm93btrl+uqDaMW7BlSBLLuIOteSapvsg/uDmK7r+RVCCCGEaJpWxRphj4itUK4a4DHLe7/o86JbICAoAJEF4WdNeNcwtMndMsHSFRCPeHooAEJ4HqKSqoRVRSXeIsvPAjxyiFSECYuOA+GBCBrCBQkbDIme4Wx7k5Pn3+76dvjNT4fXck25DnZtyE1DGFfxuHKdEdNca7zDiDjGmBBCCCHEoBmoWMNjxvtZVSIl1oYLvD8Y0FlY6XUTGnXAc0XuFt6rQUN/EI54wBCRLDNQdW05Ay8R19KWLLDQv8deeje5+Be/d68Z31n6oBM1oxQKaSGQ519xX6+Xnwbvml0LxNs1v6q3iDr7ICyS8EjuJQIuNrxSCCGEEKINBhoGaflsocIjsHv3bom1IYMFigndywLPG96hJkh7nvoJYorwO8QCJeLJpWsixBPwDpHb5pemN68ka7DhYfPZtPOAEzanfO+O5LzlzwXFz3C1vclnF9zl+rTuiX29Xn4axBXXH28mcN34fxOLqBMSaXmGhEoiDOVtE0IIIUS/ab3ACGIrJMgsHw3BloVViszyvA0bew4ecTlGyx/6jVvk97s3Ppx8dfH9yZk/uMM1vAi8xntsc//Wve4zH/9jeIppIGBYnDgLDF4EVhPeNUAYNuWtK4JzJ0QODw7HRDwVVSIsi3mI/EWf8dLh6Xn1TzPrzYWKk1xw7S+duGFdsrAAGp42cdUjri98N4rGPkINDyvXDRDRjK+mFlFnv0wGcE+4B4RbjsryEaPCODxXhRBCjC+tizWr6Ih3DU8Zoo1/zavG+yH8hbLZfhj58KOPk6f3vZJcs2pbcs6P7nEGaJU2ccldbh/b9hxy++wyGMsYtXleCDxDeZUjy4KAwkAvU2CiDAhCBChCifN+/MB7rXlZ8JxxDBMfgIeHMEuOm+WV3HXgdTdWTrro9uTcZcNbxn/ezXtmc9W2P3+417t88Kbh+TIQU9wr8gWbhOI5eDjxtpEfx+Lbw1SVdFQYx+eqEEKI8aV1sZZe9NpvvJ5eQ42qkHjd7DNZIZRd5sCrf0yuvGuLm9X1jYPPLvh5MueK9cnE4qnkC9dsSb50/ZPJl298xhWGoPE3r/Ee27DtqT/8pDHCPi+bfNQdo6vghUBAZYHQsdLrTUF4JUZ7U8Yzhjn7JG8Mz8263W+3vjYXxyOM1O8DAhThwb+IOAv5C7Fw5ZQbI3MufzAohIahMebpA56QWFi7jvvkC1yEGtetLS8Y+7dCMgj5ogXNRX3G/bkqhBBiPGldrAFeMvLOfKHGItm8ngbxZtsg2ELbdJVX3/6zC7WxhYr/z3dXJKddvMoZCBgMIeM0pp1707MuNOz0S1a7fZqBgXEeqpQ3aBBqFGnIwy+93hSEJXJc32gvAyLJzstC3g682R8jHK8ZAjbtHeQcyF1DKHJOeX1j/NnYm7tka3Asdbl98bonZsd3WaOZe5YOfcTzFbqmTcK+EfLkLZJnSMGbusVlxCfRc1UIIcQ40xexZuA1Iwwy7U1Lg0gjPLJou67wztFjybWrH581Jk66cEXyuSs2JvNubj4cjX2Sl2ShYhwTQwaDpivgOUNY5HmiEB14Q5oMVWOfGO15OXMh8Mwg9PCSEGZIRcC2whxDcHyOna48SMESriPGP163mKUKVm3e48bFyfMnk7OX7giOoS62L9+4050z5z658Zleb+LBS0sobPq+UdGxX4uocw4s18C9ZDz1S+iPKnquCiGEEH0Wa6PIlt0HkzMWzBiZzM4SgtaPnCGOMWfRw+6YHPvU+SuTDTte6p3V4EEwUWwkD7/0elNglBO2WHRshBAeKzwvnAN5dKHCHW1j+VWhkFAMfkQaIGxji6iQh+MMzu/dMRT5axjKVv1x0c8293pRHsJE05UygXDFfi6izjhi/DEOCWsd1NgaZvRcFUIIIWZoVayZJ61qKCOfpRJkV0Mh8QDMGBO3JWcsXFMrJKdq45hnXjpTCZD207VPThulgy96gFcBQzUPjGeEUl4eVhUou44IS+8XDxWhcRSHwAuD12WQlf0IoeM8Qvl9CDP6gKeIAidlRC33n5wvxsNpF9/T8cWy9yanX3KfO1cqWh77W/W8QK5ZyLuGgOeeE1Lab7h35rXF69Z09dBRRM9VIYQQ4jitijXEFrlnVddJm5qacp/vWul+DEor5sAMLLkToR/8fjYWRLbZ4PnL1ifvH2tmza864A0qKryAoEKINO31QCxiuL84bSxTet2KQWAwE3rZLy9LFuYBJN8pBJ4g8xKF1lYrgvtP+XLGA/k9XfSw4VGbyRe6zVX1I+ytLlw3KjamQRiTVzaoRdSZKODYTGDYfW8zl24Y0XNVCCGE+DSdFmt1P98Gb737QfK1q9e4H29ybL50/fbgj/wg2tlLn5o+p5lKaawtNOgkeUL4YrwZiDW/9HoT4GX58cbDyfnT12Xh2t+74hNdKfxQlFuH0LTqhniJEJlVwui4/4yDmbF6hxsfoXEziHbOT3Ykp3z/TnduCLWmxir33fL80pATyHuDWETdh/trS0EwidD0EgPDiJ6rQgghRBiJtRIw82sGBaWfqSYW+nEfZMODgifFDIv3Pmh/segsYioYQqj0ehUQNHigEH94UchJu2fHEefJ6EeBiVjw7uXlUCHkTLzmra0WA/cfjwDjwXkrrt4cHDf9bJRQt2Ii375+XSMeNR/CDkPeNSAMEZEUm//XJoxJwnIpgMJ4ZXIjryjPqKLnqhBCCJFNp8Ua4Y9dEWvkK1x866aeQXF3p/OAODdylThXcpcGmWtRtDaY4QuUMuB5onoja7thhGOop/OC8GKkF5oeFCxwTf5Ulnjk3BGXdq6x1y8P7v8Na7a78UAjF+fcm3YFx06bDYP3rMsfnD0Pqu21MTYpHsNYyAozZJwRItslYUS4MF5oJjcYy3j/ujBe20bPVSGEECKfRsWaFRSxZjlnrKnmvx7TJicn3Wdp/D1olq172v1IEw7TxZnfdMMwttAdqgMOiljPECIlVBwiCwslw7hF0HCcrM9i9LL+GrlfgwQvCkIsL1fJL7gS65mM5aEn98+WQT95+l8WB553c/vGMcegLPpJFx0vi37/1r29s2oHRHveUgd4XdMLkHcBxjBiEkFPgRk8hBTMGVX0XBVCCCHyaVSsIbJMYDXVWHNt0OutYeTy40wYWZfyfooa62xZuNndj+zu9aa/YAzH5lxllV43EC+EiiF4rEhDrHeE88A4t1L4/QYBhvGdXkvNx4qtGJxrzNpqZWDdKMrjz4znaeE0PT7+afGmVjwa5y1/zi06bMYtjQISZRe8rgLjIs+7BoSj1llEvW2oVMr9R7BznniQR8nbpueqEEIIUUyjYg1RRejiwoULnTctJL7KNPYx6LL9b7zzfnJqb6FUcm1CP95dbnOXbHXnjjdjUInxeDliqhmGSq8jsvCaIeQwWvGm4VWrAoY7+w+Vym8Tq0xJGfcsMMIRoH4IZ5m11cqCYDpeeW9mwWHKpJPTVmfRYT7L9+TMS38x60mjETa2/+X+CmWETlYRF+CaX/7gH3K36QJ8HxizhAozjgilzRP9w4Ceq0IIIUQcnc5Z6wJX3rXF/SifddkDwR/tYWhzFj3k+kBuyCBAgFBEIQZCJvEo8Rlbn4ocHrwKsSGSeVi1wDzh1CTkT+FR4/zzIPQNY9wou7ZaVXYdeD1ZcMvG2fBIaxRT+NwVG514m7tkm/N8EKaGt4zG37zGexjbbDuzXtpMmXMa+2TfHGMQIM6/PrnP3YMs8PgikosWUe8KiDTWB0S0MV4QcU18L/qNnqtCCCFEHBJrOeAJ4Mf4pAtXDkU+RVbD23FybxZ7UIYz1e6KFqDGqL5t22vJ+be9kPxgze+cN65KyfoiKN6AgMoz4psAsYAQKFrbC2MbL5rvMYz1RjYFFfk27TzgQiTP/MHxsMWyjc/isWNfXaiYR24a1zIPK+lfJKi7BF5BzpfwSM590Au8l0HPVSGEECIeibUcCN3ih/hzV2wI/lgPUyMvib5885rBLDCO0YxBmYb1sFgDjXA0vAVssyRjYeMmQUAhpNoQg0D4Jn0K9TkNoswvwlImz68NPvzo42Tni68lkxufcVUkL5t81H0X5i2614kxGn/zGu9du/pxt+325w+7z3YJxhdipihsEE8m21UNsR0kFCAhNJIJCAqT4KXtsrdNz1UhhBAinlbF2jCzbc8h9yNMcYR+VMxruxG6ZosQb9jxUq+X/QMvFmIMjwCNhYAp8IAo4V+/VDnbYji3LVYQUggqO25TsD8WO6YVgVGNke3npmFs11lbTXwShH/M9SSkkDE6rNUXGXd8jwgb5vvDUgBdWE/OR89VIYQQohwSaxl8Z+nMelBUswv9SA9jI7eIPg1qFpi8tRumXnYGMSKJUvZ4PkIUlV5vAoxbjPibfv1q75VmwEiOFYEIiXSYHrlIdddWE8cx71qMcLFqo3g3hxmqYTK2mAjge8d3rQt90nNVCCGEKIfEWoB3jh5zP75UyOvyIq1lG7PAVqGPEu79AO8YYX4Uy/j3n+1PFqz5XVSuWEzp9SbAgMWYbSrskkIVGPtZItTHRIR/PfDq8FrT3r5xJx1qmgfiuSuLqNeFPuDFxsvL96lONdW66LkqhBBClKevYm379u3Jdddd53LYyrR+L4pt6/+cfsnq4I/zMDeqr9G3NtcHIrSP4gcYxxiIGL9Ud7RcrBghA3jW8FK1DYKQAigsEVAH+ozXEKEZA3lG6f4hGtv2KI4jjEnujb80QhYIHMRaUWGSYYNxztqEjHXWHCRvM/a72AR6rgohhBDl6YtYo9DIiSeeOLt+WtmGYOsn85etdz+8lC0P/TAPc7OQHcKRmubg28ec+MAzhLGL+EmHXpGfRkhWDDGl15uCAhQIyRhjPgQFKvh8bI4QfWP7tOcQQ7preUajAuLEXx4hD8YtHtJBLaLeNs+9+sFsziiitB9LWei5KoQQQpSndbHGotYnnHDCrPDib/OWsYB2TMMj1y8oYW5rTp17067gD/MwN8KPKJlN/956t5ow8cGLxGw9hq0Zt3meJUKyqFgXS0zp9abAgEVoll1w2Eq/07dYELV41nwQiv1YW21cwWPGEgmxgpyQVLxx/V5EvZ8QpmyhuzT+bqOwj56rQgghRDVaF2sLFy6cFWpTU1O9V7vLlt0H3Q8uiwKHfpRHoRGGRB/XPbGv1+tyEFKG1wzvGcZsmTwYDOYya5yF8rrahGUEMOjTHq8s2A5vGBUcY6Ev9CkdgoYo7efaauMI9yl2gXZgXHOv+rWI+iBBxDIG8bbhdavqZQ6h56oQQghRjdbF2jAJNbhz0y73gztn0cPBH+RRaJ+/cqPr44r1O3u9jgPj7dbHZow5yoOTo1VlPSc8SnjMYiGPK6YMflNwblRzLOoboXJ4wsoWJ8EgTn/G8vkGtbbauGDetTLVNtmWCYayHtdhhUkEQkbJa2MiAs957ORFFnquCiGEENVoVazZoti0YYFFgPnBnVg8FfxBHoX2has3uz5edffWXq+zwQuEsMBoQ5jg+akrKDB6MX5jMe9aP3O58CzkCUSM/qs2HC4dokkf6HtaCGpttf7BJEPZcFMLFRw3MY1n0SZo+D4Q6svYL4ueq0IIIUQ1+iLW+l0gpA4LV065H9y5S7YGf5BHoc1dss31ccEtG3u9/iR4eQgHxLtEmCOLRzctlAhFI0csljKl15sAMUX/03llBiINsVbWcKUPoVBHra3WXxBrZXPRCPeNXT9v1OCZQGEgvrdMNjCBE1v1FPRcFUIIIaohsZbi29evcz+4Zy99KviDPArt7KU7XB/9RVwxQJk1Z4Foy1l59vDR1gxTwqzKLEaNeEI49jN3iNAvvCkIVx8MVYx9DNgy4KUgBC/tVdPaav0HYcy9KHPN2Zbw36YXUR82qPqKcGXMcj1inhPj+lwVQggh6tKqWDt69KgTa5TtHxbO+dE97gf33JueDf4gj0KjGht9pK+v/OlvznvEbDleA2bP+7H2EuFkGHtlBE+Z0utNQcgmItG8gIQrEhJaJYeHcw8VIkH8aW21/sN4L1MYBhivVfIURxEmHbh+VHflO8JzhImHEOP2XBVCCCGaovVkMltfjRL+wwA/trTQj/EoNfo4b9mu5F9X7kuWP/Za36ot+pADU8ZYZva+TOn1prA11Dbv+5MTmFUKTXDOWZ4cra02GPLuSR4IdcRJWaE3yjDpY2ssEh5MXqB/XcfpuTpn0UMjvdyDEEKI/tK6WGOdtGEKhRynGeB5P306+ZcVLyTfvfeAq/yG56ofXjWDULSynjIMZDwi/eaXz76dnH/bC8kzFfPK8MaEctIQDGWLXYjmYPxVWS4BcY2A7/fEQdfB24ZQ4boi3Mh3ZXJjnJ6rX75yrQufJlS07ESAEEIIkaZ1sQYTExNOsLHmGqGRXWacciu+sWRd8m+3T4u1VQeSZw+/39oaS1lgyGDQZYVOheAzeEP6WYzDFkde+sgrrsBC2Vw1zjVLkHHNtbba4MBryr1N5xHGQI5nVU/rOMB1Qaxxjb568zPJmZf+cvrZ83TwmTQKzc9Z4xnBc5TnRZnnmxBCCJGmL2INgYZnDcF2wgknJJOTk8nu3btdAZKY1s8QynGoWvbF6x53fZy/bL3LvUH8WJU7f40lZocpWd5muXJytdY++3bvf3FUKb1eFa6HXQdAXFHRMXbGnO34fKjyJQYd4lhrqw2WrAqdMZDjyfen7jpko8yHH/0j+faKp5PzbtmTzLv5t8Fn0ig0/7lq8CxFrCLshRBCiCq0KtYQWZazVqf1M4Ty2tUzP7ijvB7QhLceEGIEwfDfDxz8VJU7qhda1TdbY6lpCCfD2C1LVlhhkyC0ELHk4hi8RogXHoMYCNvMCvXkPa2tNngspLFqCDBjgXFS1uM66lA1kskYnh/fXPnb5Lzle5NzfvKb4DNpFJr/XPXhOlDAiQIssZM8QgghhNGqWMMrFhJfZVs/xdqdm2byDuYsejj4gzwK7fNXbnR9XLF+p+sznq0fbzzsPECrd356XTGMUFtjiWIY63a/3agnAeFVtiQ/Qq1KcYgyIFBp6WOYt41rkgehdZxjVt8Qcf0M5xTZcJ+rVnhkfCC62ce4w9preKHxzPOs4Jq+8Ppfkn9b8Xxy7rLdY/Vc9cF7TuGVxesPyQsrhBCiFK2KNcIf0yGNVVo/wyAfffb37gf3tItXBX+QR6GdsXCN6+Pabc+7PiMqyNthvaSiKnfMElvVNwzUJtZiw7i79bHXev+Lp0rp9ViKvCVUz2S2nP5nkbeQt9ZW6xbcT+5HVe8a44QS9lmLqI8yVlQEMYKHku+y5bwiUpjY+I879yVnXfbAWD1XQyBeeW6oMI0QQohYWhVrw8h7H/w1OeWile5Hd97Nu4M/ysPcyBs56cKZ/r317nGDAWGBpwcxhsFVZExgoCGUMFAxPjBCmFWvAjPNHLNskQfOsQ3vGnkmGJhFM+CEiSJuQ2X3TQBnleTnemlttW5BPmIdscV4wZuUXkR9VMFjjDDju4vH6PED733iO8zfTHjc8MjL09+nA9PP1dvH7rkagvxVnplEKAghhBBFSKwFIEGcH11yEEI/zMPc5i55zPWNimU+GFY240toHl6G2Cp3rLGE8ECchNZYiuGaX71caW0iBCbiqik4d65D7LpzbE+/08IOMYbxn4XWVuse3POvT+6rFabGdwbxEiooMwrgEWZsM0nChAaCIzRJw/ffwoh/uGYmv3Qcn6tZcM0Qsnjeq3pzhRBCjAcSawEIY+GHl7CW0A/zMDdCkegbuXlp/GIYhCZikJUxXBF8tsYSAgYvRWzZakQPQq8sdUqvp7F94TErAwYreXcWMonxhdjNEnwI4n5VsxTlIMS3rscToYZgYxJjFGBc27OBcc01wgOfB9cQMfL0waMu1xXG9bmaBYKWcGtN3AghhMhjIGLtyJEjyfbt292C2TT+7hKEsfDDS1gL4S2hH+fhbHuTk+fPhCIdfvPToVoYDwg0y8PCKCPMMStvKw9bYwnxQ4gUIi5PUPEehmCVUEpmp+uuVcb54lGrWvCDqpl4B7mGiFT+n4XWVusuTE4gtOqujYW4YTzV8dINGkQnFWLxNvId47sRMynC2Mbr9u5fPnL/Wkj1uD5Xi+C68pwsKlgkhBBiPOmrWEOUhUr5p6s9XnfddQNfQPs7Sx90P76jFLJjoTrnX3Ffr5efBiPTvD4IDwRInSp37MO8ZogxBFxWeCUCp0oeB7PSGDtVw4kwqBGpdYwl+kkfb97yqjP2s4x0hC/va2217oJXKC+ENRYqq1ZZRH2Q8N1ksgGhybkjvMqMVQsjRuz6nnpjXJ+rReCF57nLuBum8SKEEKJ9+iLW/EWxQ80Xa2xrr09MTAxMsG3aecD9AJ/yvTuS85Y/F/yRHq62N/nsgrtcn9Y9sa/Xy0+D6MBoMA8ThgP/b6LKHQacGYIYcRhz/kw94YfMxFehaul1+tdUFT/29a2f7U/+58E/9F75NPQ5q0Kk6AaIk7ww1jL8dPMr7n7zveoq9Je8T74HTHowoVIlhJMwYq4bIZJ8r5kASYdLjutzNQauGWKNZQ+yJrSEEEKMH30Ra4guE2B4zSjHD/ybFmtAqf4TTjjBvTc5Odl7tf9ccO0v3Y8w6+eEf6iHp01c9Yjry1cX3z9tOObP3CLUEE1mYOIlwohrqsod+yXUEq+d5cCYcchxi/JhQmBYs68y3jXOA0Mag7oJOAfE2n/+/MXMcEpEatZ7ojsUFYiJBQOc3K2uVf5k7DMOmeTA08t3gEXvq4pKxAXPCAuhxiOXNSkxrs/VWJjQ4VpWKbgkhBBi9GhdrBH6iOhCfKXXS8sSa2Dv0chxGwS7DrzufohPuuh2t6Br6Md6GNq8m/fM5lRsfz6uiAfeNIwGAzGFUYdB1yTkqGEY421jZh8Db8W27HyvPDCuQ4t6Z0HYJeKpqoGahuPTFwvLTItOPIu83tTxRHsg+hnvTRR+sLXGmqxaWhX6g3BkYgMRyQRM3WqEFkZskzkIVL7PWd65cX6uxsJ94poykaXnhRBCjDetizXLUZuamuq9cpw8sQYWOhn6bL9YuHLK/RjPufzB4A/2MLQ5V8yUzP7ujQ/3elUMxQUwFnxDAaGGAdtWlTv2/+NfvezO+bZpwVbWw4ZnK7b0OrlxGNBN5YeYQLP94WGw3B0DIUeImRgO1j77dmMhq2nPUz/h+0B1VyZgqDzIhEYTIZ7AeCe3zZ8kifFKjutztQxcWzyfXN+6BW+EEEIML62KNT//LJR7ViTWEGm8T+jkoHj17T/PLpI9d8nW4I92l9sXr3ti+txXuPM/8Oofe72KA69TOvSRIhyIkBhBVJX/eegPrgodhiWGCh6J2Nn/mNLrhBdhOFddxDsERn26wqMtrm0Cjv60JXSbZs/BI67U+vKHfpNcdfdWZ5AS6nXmD+5wjWIKvMZ7bHP/1r3uM02FgnUBPESMkya8a0BOV1PeuiI4d8Y5ocYcE/FUtNB9WZjIYf98Vw2+pzH5fuP8XC0LzxXGYdNRDUIIIYaDVsVakRir+36/WLV5j/tRPnn+ZHL20h3BH+8uti/fuNOdM+c+ufGZXm/iwbjDSMDw88E71GaVu8cPvDfr0eAcyKcxg7NoDTREJNtmiUn217R3ECMcIZa+ToBwZOmCva++765ZV/nwo4+Tp/e9klyzaltyzo/ucWOmSpu45C63j217Drl9DjsYyty/prCJgrYmOxiLhPcyxjlvvkuhcdkEtz72mjuG732nUE9sft64PlerwHOPSTI8mAqLFEKI8UJiLRIMUH6cqWI2DHkW827ePVulbNHPNvd6UR6MsbTHCAjPQVC1YThgXGJs+iXD+ZtQLjxVVEvjnPz3fTAWQ2FYhKKxX0I8m8QqW4bg+nCd5t/XjZylNHgFrrxri/OWMVasfXbBz12Y18TiqeQL12xJvnT9k9NG6jNufSwaf/Ma77EN2576w0+KPPZ52eSjrXse2oSxiJHcpFeK8EpCEpua7MBDzD4JW+b7QYhvk17jEByP76Hfh6KJkhDj+lytAs87nse0tsS+EEKI7tEXsUY1yBBFYmz37t2dEWuEdxH2xY/0aRff0/FFXfcmp19ynztXKq8d+1t1w81ysdKz8xhpFAQh7LANmLUPiUTAe4AYwzDE65Y2pDFq0qFYGDf0A09Dk4Ry+9K8d+zj5F9WPJ+s/k395QGagjA0QhgtFI2QrtMuXuWEF0IsPK6K27k3Pesq5J1+yWq3z5l93+ZylKosGNwFEOII8iZh/LIuX9XJDr5/dl5WUbXI69wUfIdCodCcQ5kCPzCuz9WqMF6sIBPPQSGEEKNP6wVGEFu0KjlrlO3n/bVr1/ZeGSzvH/vQ5e3MGBarOjkTzMzvjKF8mwtne+dofQMOzxAGQhqMNcL/2vAYIcDwPuRBfgzHZobfPApmQPpFDjBs2QZvQNNwjkWl+DGqye3BwBp0OW7Gw7WrH58VaSdduCL53BUb3bgJjac6jX1Snt0q5nFMBCJCcZjAQEaQN5kzxD4RWoQsloHJAZuo4HvJItRthTmG4PgcO70OGBMjZZfOMMb1uVoHxiKTTzzzhBBCjDatizVbYy0kuPLEGuLO1lrDw9YV8A5QXIEf7ZPn35GcvfSp4I/7INo5P9mRnPL9O2cNiqY8GXjXsgwxjDbea6PKHUIwthgDXgXL1SFEc8cf/uzOi4qShA3hqWsaRFqRoASMcq6PXatBzYhv2X0wOWPBTK4NXi8q8fXDMOYYcxbhPZnxtJ06f2WyYcdLvbMaDhDcMfe6DEwiMMlAeG8eCCE8Voh9ziEvBLhNbPmOUEgoArLOZMg4PlfrQqirLXdSRSQLIYQYDloXa7bOGg1x5pMn1hYuXOjeo/R/13jvg78m85fNlG3GAJ24enPwR76fjdwhS3r/9vXrGp/5xRgLedfAinY0XeWO45UtdY8BTMVKinn8+x37km9Mn9fi9dXDzbKI9bak11bDM4HR3VTp9FgohDAzXm9Lzli4plaoY9XGMc+8dGZBZNpP1z45fV2Go3ok9y/Gi1oWxgfjIb1fjG/GMcY444fvwSAriVoYccgzzPeePtT18I3jc7UujEtydHkWNf38FUII0Q1aF2uAGDPBRhl+Wxw7JNbwopk3jpYWeF0BI/OGNdtnDU+M0HNv2hX8wW+z4bU46/IHZ8+DMLM2DGDEBYIsnadi4HnAmGuysEHdRaRveey15PxbX0i+MbnPhSHi3aq6rzSxeUwhwclyCHhU+jEbTl6NrWnlDODFU8Fx1M+GEW5eNoxzwuCGAfOkNjWGDLzCjPMX3/iLG6N4hvmukY/JZEDTxyuLeQCzQu7w7GTll5Zl3J6rTUE4LF779FIrQgghhp++iDVCGn3BZs0WzCbc0Rdo1vDKdZ2Hntw/m/9z8vS/VMWbd3P7SfIcg3ygky46ng/EWldtgnctryw3oVrpCnF1ufzBP1QKscRoIYxy7a63k//dcNiJK7wUeAAQUHUWmcWDwL5jKgSyXcgjQolz+tamIf7Wux8kX7t6zczYnD+ZfOn67cGxNIhGmBvhbpwb4W/DUnwEscZYahI8Ij/eeDg5f/q6LFz7ezd2uxLWxvjMy61DaBYV2KnCOD1Xm4IJNcYnz+kmn8FCCCEGS1/EmkHemuWh5TVCIM37NgxQMIEyzvy4ux/4acP4nxZvaqWy2XnLn3PV9szQpeE56Ud5dLxmed41wBtQp8pdGgxXPA1lQNzhqSBHLF16HeGE4GQWmvNkRrrsueJFwJtQBMfMW1uNfvkLCjcJHjUTapTUp0pjaDwNsuG9oKAE54hgIwyu68RU/4yBnDPGEcY1gp6Jjnt2HHEerC4Z2nyfGetZ/UXINS1ejXF5rjYJYwexxqRZv0OthRBCtENfxZpBaCPCLd3wpB05Ui5HqUvww3485Gym0h75QYR91am2x2fJnTjz0l/MzvjSKHm9/+X+VgND6ORVsMOow2NUtspdFhgfCMRYTwMFRdJFPEIhi4g48m94HWGHpytd4S4E58P2MfkhGE15lTI5B65VVi5gVQjXuvjWTW6MnPrDuztdDp1zo2S7jecuh5oZVQUK95vJAUJyGdOMj7R3lu9NeqHpQcF3Am90lnjk3BGXbZ/rODxXm4ZJLp6Dg64+K4QQoj4DEWujzq4DrycLbtk4G8ZjDS8CZdIxMuYu2eZCwfB4MKtL429e4z2MCLadWdfn+HpV7JN9c4xBgFft65P7cmdt8RrEVLmLBe8TxRaK4JzwomEQ+2BM4g3JCltEpJFXhgjDEMfAwbAOgbCK8YZh4GIsFVXt43pybk2u/7Zs3dNurOAl6KJHLd3wsJlHg0WSuw7jiLGSNUbSWKVSxgNCjHud9VnGKh7fNqqXloHvG9/hPC96GwVX8hjl52obMKHEs4UJti6IfyGEENWQWGsRQtE27TzgQnnO/MHx8Jqyjc8ys8y+uhAqRsgWXoE8rEx9WjhVgdAzZvjzMIGYVegATwjGZR4YNJwvxjLnnq7Ah3eP12PCizhebPhmkyX9yfWZGTcrnIEaEkddbGcv3TEt2Gaq7t39SHeW6sgC0ZVXVIOQYUrZMyZpFOeILb6D0G9rXcAYEGBMeuR5mq3YyiAY1edqG/DMIoyVcOwmiz8JIYToHxJrfeLDjz5Odr74miuhTrWzyyYfdeE28xbd64wGGn/zGu+xcDHbbn/+sPtslzDRUhQ2iPhgOzwLdUBE5RmPFk6YV/yEfZTxBFCAhDAwjotQRHzduf2NaI8HHroyhVHsWhVd0zzeeOf95NTeAtR4EEKiqMtt7pKt7tzxcnS94Ahei7R3DZGF1wwhx73Em1Z17OPRYv/9DmOzypR5Ewd8lxCgWZ7qfjJKz9U2IcqB+9rkwu5CCCH6w8DEGhUiyV2LacNUbGRcIBwwpsgGxiZGQp3qi4BwwqOXBsMRDxaNv/Oo4g1gn4guKkoiKJZPi7WifLWqSw5g6BO2lBd6lseVd21xYuesyx74lBAaljZn0UOuD+TcdR3GP94vRAueZvLQyEfDOxsbIpkH46wpj2sMWWHEaUI5oKL7MI64vzy7yz6bhBBCDI6+iTWKh1DlMaYaZLqFFs0Wg8W8azGFNiwcLKtQQQx4nDA00uBNw6sWaxwj1qoUh8BLglcNQ4fzIKyIvJ5Qn9im7GLeBp8t0x+DggiInJMuXDkUeWpZjaIPJ/e8g13OH0LY3LbtteT8215IfrBmJvy2KD+xCkwUMN5iQm/rwAQB39G8gjjAuGRCoa63XAwG7jOeX1rVSSEhhBD9pS9ijYWwQyIstkmsdZPYEvaA5wEDoc6MLuGI5K8ZHB8Ds4yRXKX0OoYywtQqUvJZwonw5uFNSYe7Za2tFgsFTCiZXgbCvBA4n7tiQ1AEDVOjPDt9+eY1a3u96wbcf6rsIabxnCLIl0yPfwR2myCgyo7zMjDhQJ9iJhjKfOdFN+H5RZQCkwD98toKIYSoTutijZL8JrrwqiHc8LKFwh2zmsIguwmz7BitMbkrGAiItaLCJHngybJKjIRqVfU4lC29zjlnGeTMTlM8AoFGUYifbX89ufgX+cVQiuBacY6hsM8Q2/YccuKGior9WDi47UYFv1O+f6fr04Ydn/am9hMT5ohnhDn/4u3idTAh35aQMhBSCCo7blOwPwsjLoLvO9+5GG+66D6Ma57fTVXtFUII0Q6tizULe5yYmHB5amK0YNY/Nn+FGXw8BFWr3GEQYzDveeUDZyCzploVEJeIq5hQQyskYV61PPDafXd6v/92+wtO4NWZteZ4XCs8OUV8Z+mDTtiwqG9I/Axjo0DKIL1reEYRSNx7RBITBVljgHudV9ymCRBVeLSaXkR98omZMOIYEciERZ3JFtE9yK8lpJuxFfOME0II0X9aFWt4xMyrNsyLXYtsMPIIK4ytDGfFN6pWuaPQxzemBVuZSosh8PLllV43MGJitgPEKCLy9ff+7marEVs0/q7iecFrgyfDD/1M887RY07UsFBwlxe/LtvwrtlCxa++3Z+KiNwj7jV5jVx3PJsxnltKojOJ0HYOEOMLw7qpsEsbozFGOtswtqt4skW34RmOaOc5Lq+pEEJ0j1bFGiGM5lUTowshhRiRsZDfheFX1vOEMfwfd+1P5k8bmHUxj1med82qp8V44IDrkA4n8ysFEkIXK2oNzoHzzCrpb+uqnX7J6qDoGeZGVUv61ua6a9xbQmoR5dwj7lXZewR41jB424bvAF7huouo02fGFUIzBqqx9qN/YnAwgcZzuUoBJiGEEO3RF7GmAiGjjXnXYtcwg5iFd33Mq7DqNzOhaXWXAgAMdPLNsiib28b2WR4/PBOEjJLXhrHNcWM9MRjWfCa0/fxl652gmbh6c1DwDHOzUEjCPJuGEFrEB8YpXlbED2OsKtybr0/u64vnie8MwrKKqAQmAPh8rBeFvvXDcygGD2OLZxSTFnW+D0IIIZqjVbFG6KM8a+MBgqLsGmYWhlUUIogYZP0qy9fBi9FEKBjGKsZ6KAysbNXIMmur4VlkGQAMYDxxJPoXfQ5xRzVM34A69re/uwWkETTn3rQrKHiGuRHWyVIE9O+td6sJEx+8SFxHC08ldzLWsxQDYZP9yulifDJ2Yyc7DLbnc2UWR0bU4lkT4wHPGMYxz3OFvQohxOBpvcDIiSee6ASbiouMPvy4l81Fo+x9UYEDhI1f9h+RhaepCTBKQsKPviBAY2EfZddWwyiicAUeQ7yM7CNPPHAd8Abaddiy+6ATMqddvCoodkahEd5JH9c9sc/1uSyEOeI1Y/wgptPLLDRJv/O6KD7DhEKsx4vt+N6U8RbTl6wJDTHa8Gzi3pd5DgohhGie1sWale6nZL8YbQhtLOONArb1vWZp8H4QlpMOyUFM1am2aISMUfpR1kuIEVxnbTXC8hASnAvXwy8Pb/D/qzYcnhWFd27a5YTMnEUPB4XOKLTPX7nR9XHF+p2uz7EQImjeS64nBmds7mEdENwxZfCbAm8ekx1FfeP7w5gu65HOmswQ4wHPJZ5tRDOUea4LIYRojtbFGh41wiBNsMnDNtpgOJZNUM8yJPGI4A0JeQ6o2ocx3gQYpFZ6HYMEwVmm2iQhaWUKrOSB0c31I+SRvhN+5ufnca0Qr8x637BmuxMyE4ungkJnFNoXrt7s+njV3Vt7VyAbhDdjCOOS8cQYaXv9szTmXetnVT0K1+QJRBP5ZUM06UOZAjtiNGFMM+HBM6nJsGEhhBBxtC7WAIFGkREEG+uuTU5OJrt37/7Ewtd5TYtiDw94NMp61wBBhjgxoYcAwiuSlZNja641YUhigFgBBY4fu26cgVeQ4iFNg6fOimBgbOMd4rpyvhjRF614zAmZuUu2BoXOKLS5S7a5Pi64ZWPvqnwSxCvhgEwSMH7wOvZTKIVAJBKu2i/4DtD/rLwyRBrjp+x3ssyyFWL0Id+T71iZfEchhBD16YtYo9AIXjXEWpWmapLDBWKnipGHkY1oQjDxb1G1O2Z765YwN2ZKr7/uvDJlquwhFjjXNj04GOPkAnJdEW4IkqcPvpecf8tvk1N/eE9y9tKngkJnFNrZS3c4seYvjo3owGBEJHPt8SyFwkYHBfcLo7aJMN1YmGigaEp6EXU8jXgZGadlIK+PSRd51YQPz0bLr+3K900IIUad1sUaQg1vWkiExTaJteHC1garYug9fuDd5PxpI/3hPcViD08ThSOaAGP331buS654uNz+MI77maOEpxGxhmg776ZnknnL9ky33UGhMwqNKpeItXN+dI/zNOI9wljEk0QoaFcLX+BpLeuhrQtjg++dLaLOpAeTD6Ew4iI4d/NyC+HDeOK5S+t3mLEQQowjrYs136NGZcipqSkX2ihGmyohVMz+4x24bvqzMVXuzIPRRB4Fx/7a5L5kScnwNURDmfy2pvjwo38kE/87lZy3/LmgyBmlhlj78k9/k/zrtJhe/thrQ1FOHK8DY7iMl7YJbA21zftmKvmVLe0PVUOZxfjA2LCJk356kIUQYhxpXayZUFM1yPHCQhpjPR/8+DObT0VEwHuEECoK3yKnK29h61gI67l+6rAzcGPFANshFvtp1FKdjZBNzvP8m5+ZFmt7k3m3jK5gM8/al3885cT0d+894Aqs4LnqqlfNwDPFGO43v3z27eT8215InimxSL0PYZNlFrgX4wsTVTyLlNsohBDt0apYw4NmYk1VIMcPwgPTFR6zIO/IX0OMf/l/UYghuTV44+pgFfysmiDnEgNl062MfpvgOWQBcUQKYW2c497XPki+snxPcu6y3WOTs8YyDtzrZw+/74pmWL5av71XsTCG8VD1U/jY4uxLp783VCgtm6tWZdkKMd4w5hhrPK/LjjchhBDF9EWsKedsPEH8IIKKPCCE01AWOv1Dz/95PavKnYEBX2ehYwSXlTU34RZTUZAQoDprq+VhRUWo4ocoYZkCEyXkidDnr966JznrsgdGuhrkF6973Im1+cvWu75zHfBWIYS4V3jYELFcDwRt13JoyKvsl/jheth1AMa0PwFSBNvxect5EyIWnldERTA5MehqrEIIMWpIrIlWwWDME1sUicjLT+N1vEnpKnc+hEESDlkF9o8Y8o8fU3q9ybXVfMj/sMWcSeCn2qVfqIW/ESs3PPJy8o2Ve6aFzIqRXmdtIrXOGoKC65JeRB2xbouK443tUnnxfoQVcl0YF/73gNcILY71/lZZtkIIH55XfAdVnEYIIZqjVbFG6CNijcIiYjzBu/b1yX1BMYYBi3eqqAgC7yNesmb8CROsWn0SA98WxDbYD/sjPyyLJtdWI4yI0EZEK54NxGeoaArGN0KEhgBY8sBvnZCZs+jhoNAZhfb5Kze6Pq5Yv7N3FY4XoiEMNQ3vMQGAkEbkcy2zJgL6BeO87YIdNi7SxzBvG9ckD8Y856hiEaIuRBvg7WaiTmGRQghRn9YLjCDUEGyqADm+MNufFkR4QhBEscahLZKdFXaIt4WQszJYmGYodC6v9DoGCOdSJ+SOfZgng3PgGuWJQ+Aa4j15+uBRJ0Yeffb3TsicdvGqoNAZhXbGwjWuj2u3Pd+7CjNYblbeDD7Xk+vK9cVTOsi12LhvbXkbiorxMM6ZFMmrWhrjTRYiFsYi+aRMKjH+hBBCVKd1sbZ9+3Yn1iYmJlRkZEyxUEMMbMBThvFYVlxh7PK5kKeE8BsWyS4DM79ZBVAw6vE0hIpX1FlbDdGJVw5vI8YxXpcYjyDGNB6Sd/8ys/gx5/XeB39NTrlopRMz824evbXWzrtlT3LShTP9e+vdT98HxBjjqqjACNeXsUP+I+OHe97Ecg9l4Bzb8K4xqcB4KPIe2uRIKJ/IPMnKNRJNg0eX72jbYcBCCDHKtC7WwNZaQ7AdOqSciHEErxDiCKMS47JqCCGhb6EqdxicGAWxIW8Ypnhc8oqfmOcrDV6MMmurIU5tTSLOHeFVxiuHqMWYRuymz4nCG4gZcrtCgmeY29wlj7m+UQkyC4xA7mPsemJ4ZhmLXE+Kt3Bt++Vt4741FToLnDtjKtZzYeMo/R1BvPLdFKINmFQhJJnv3aA820IIMcy0LtaOHDniQiApMoJgM9G2du3a6IZ3Tgw3iJNv/Wx/svCXB2uXu0+X+TcozBG73g+fL9rWvGv+rDCGMQZvkdFBfzHM8eawPX2uUjmSMFHECAaP5RX54ZKEByJoCBcMCZ5hblS6pG93btrV620Yqh9yXWKFOnAtqbaJgOL+IKbN89sW3EuOxbHrYvsqWwWVHD5C02yyw6qfKlRNtAnPQyZHmOjqt1dbCCGGnVbFmr/OWp2mapLDD+Lm+26dsAO9V6qDscuPfjoPDgM2pkx6GaMZb4S/z7y11egjwo4QSbx8iEqqEhYJuyzwFnGeVtkwlFdEeCCChnBBwgZDomc4297k5Pm3u74dfrM414vctNDyDzFwnbmnXGtyHxFxTQiqEDGTBEVYGHHV0DKK6hAyzLhEpNpC9EK0DWsl8j3T8hBCCBGPxJroC2ZMI2KayI1hpjYUTomHpWj/eFPKFHtArJlhjJGc9pBxPIQjHgpEJDlteeGVMeAl8pcsQDyEjg3fWfqgEzWjFAppIZDnX3Ffr5f5IDwQIFVzCYF9IM7xAHAvEXCx4ZWxMFYwVquOD8YFY7youmMe9JM+3rzl1VKhw0I0AUKNZxnht1UnsoQQYpzoS86aGG8IU0NYYaAys5r2DlXFPE9+/hgGQNrj5oOhgLFbxkhAqHH+v335/dm11TBw6RdCDlGFx62pUDK8QxzHL02fl1e0aecBJ2xO+d4dyXnLnwuKn+Fqe5PPLrjL9WndE/t6vSyG68b9KFpEPQZCIi3P0MR9U942BCX3syz0L2aR+BjYF2HJ//PgH3qvCNE/eH4yscWkQZn8XSGEGEck1vrInoNHXI7R8od+4xb5/e6NDydfXXx/cuYP7nANLwKv8R7b3L91r/vMx/8Y3rVqrKiB5QNh8PL/JrxrQEij763jOHhFssQYxjxhbmXBsMBQRjDhweGY/F1UibAs5vXwhVlMXtEF1/7SiRvWJQsLoOFpE1c94vrCd6Ps2McIZHzlLaJeBu4HkwHcc+4BHuIquYc+3Ef2Vca7xnkwyUFobRNwDoi1//z5i5XDKbvCOD5XRwHGtE2IlM29HFY0VoUQVZBYa5EPP/o4eXrfK8k1q7Yl5/zoHmeAVmkTl9zl9rFtzyG3z2EhLaQMcnbIDWoKxBcGuoVz4QkJVWvEKPXzz2KhH1dvPOyExKLpfbNMQFs5TRRJ4dr4YhODJs9bCLsOvO7GykkX3Z6cu2x4y/jPu3nPbK7a9ucP93pXDsQU485y/ZqCwgh4xDAuEe6EIlbJkQPEeGhR7yzIK2Nc++OiDhyfvlhYpl+0puuM+3N11OC5zORF3VzOLqKxKoRoAom1Fjjw6h+TK+/a4mbK/AfuZxf8PJlzxfpkYvFU8oVrtiRfuv7J5Ms3PuMKQ9D4m9d4j23Y9tQffvIBzz4vm3zUHaPLEKLID3DIYLb8qya9UoRXIsQwnglZS+cuYeQS/hhbch/DnH3yGUIgr596JfnWdH+a8tiE8PtgIEBj84oWrpxyY2TO5Q8GhdAwNMY8fWB2uQ6MO65bXS9YFuzfCskgpMqKHTxbrLUXc1+p4MgYrCoM05hAs/3xneD72HY1zLrouTq68H3g2cd3qqlxPkg0VoUQTdJ3sUYpf0rxs/YahUNok5OTvXdnYJthXED71bf/7MIXbKHi//PdFclpF69yD10ewiHjNKade9OzLjTs9EtWu33aQxvjPKZSXr/BAMX4yyvikV4vrAnwFhBC+MHfPnZGtJ8LEXM8E3psZyFvFp5DGOSD00YzRm4bXjWOyzVLG++cQ6wHhvFnY2/ukq3BsdTl9sXrnpgd300YIni+Qte0Sdg3Yoq8RfIMKXgTG97IvS3ymJrXuMly56GKlJx3k4KwSfRcHQ94rhJZEFMkqqtorAoh2qBvYg3xZYtjF1V75P8nnnji0Cyg/c7RY8m1qx+ffUCfdOGK5HNXbEzm3dx8OBr7JC/JQsU4Jj8O/Eh0AYw9ZkiLCiiYp6vJUDX2idDC03HTr1+drRSJEYAxneXJo+gIQg+BhyFLnp0vyJj1xWBm/02UXk/D8Tl2uvJgldymVZv3uHFx8vzJ5OylO4JjqIvtyzfudOfMuU9ufKbXm/pQ0TG0iHobML7IKeNeMp6K8nAQemybJSbZH+836R0kpJfvQmjCAeGYDsEdJHqujidMXPG85d9hQWNVCNEmfRFriK4TTjjhUwKNxbHTYg1Rh1DjdT6Dl63LbNl9MDljwYyRyYwXIWj9yBniGHMWPeyOybFPnb8y2bDjpd5ZDQaMvHRxjDz4Ma6SQ5YHRjkegtu2ve7yigBxxXn5IITwWOF54RzYJqsqGdvZ2mrpELK6sD8McozoNFxHQiPLQm6D+xH/3h1Dkb+G8WHVHxf9bHOvF81BaBUiu18ihHFkFVAX3P+73LGFQAp9XxDujIum16NiMiPLCLbJCLwbg0bP1fGGCQom8/hutBHJ0CQaq0KItmldrCG+TKghwgiBNGwdtrRnjc+EhFzXwAMw84C+LTlj4ZpaYQ5VG8c889KZSoC0n659ctroGkwoEz+siKJYo5jtEEpNV6Mj9wYRhleKXCLEFaIIDxWhcYg4XkOAxXgt2Je/XdXS62kIbeM8QtUpOV+OW8VQ4f6T88V4OO3ie1wuRGjsdKPtTU6/5D53rlS0PPa35vOmENbcc8IO+w0inO8FwguvW9q7i4hjnDJ5YOBpY1xQyKZJYpat4FrhiSS0c1DouSqAsch3ht8I//vRJTRWhRD9oHWxtnbt2lmhls5DyxJrgEeN92hdC4fEoLRiDsxqEY8eeoj2s7Egss2wzV+2Pnn/WH9/3BAv/KiW9ThZhcamvR6EoH1tcl/ygzW/T3607g+zxSD48Sf0MvZ4GLi2tppRJTwxDdcJzwsemBB1wy25/5SEZjyQM9FFDxsetZkcjNtcpTRCidoCAUT4X3oR9X7BWOHYeNq474ghC3/ku2PeNcYF21TxqBYROzHCeTFRUGWJizrouSpC8Bzk2d30pF4dNFaFEP2kdbFmXrXdu3f3XjlOnlgDy3FD8HWFt979IPna1WvcA5Ecmy9dvz344BxEO3vpU9PnNFN9ivVa+pV4nFUcIxaMyKbzE/BM/Wjd7911+eG0YKOKYxVx5ee++WBcV/WuIRQpWJLl6UFoFnlAYuD+Mw5mxuodbnykx8yg2jk/2ZGc8v073bkh1PoxVq1CaWxF0Lbg/pJXiQHKJMKOP/x51gtMzlgbYYg2KRKLXatQeG4b6Lkq8uC7wW8M0RBNT+yVRWNVCNFvWhVr5h1DsIUoEmuETPL+woULe68MFmbT7CFNOV0qNIUemINseFDwpNjD+r0P2q2qhZcKoy5dHKMMMeFZMRBSxiwsRilelP/+5e+dd+070z/yZT1+wGcwqEP5RnjXeK+KQMW7l5dDlZdXVBbuP7OsjAc3A3z15uC46WejLLUVE/n29eta9ailsaIdiPlBw/giLBfP7b/fsS/5xvR5LV4fH0YcC/urUsyH7yUGctshaHquihh4DhNmz0QXIeSDQGNVCDEIWhVrRWKs7vv9hBjwi2/d1HtI393pPCDOjVwlzpXcpbbi1/OKY5SlqkAhp4vqjdf86mV3Lni8MMgRUfz/gV1vJfPvO1Cpyh3eODwfWcSUXk/DzDDGRpZ45NwJg2vSYOf+37BmuxsPNPIbzr1pV3DstNkwIs66/MHZ86CC2SByKxhn5IQNyuALcctjryXn3/pC8o3JfW4s4/1ragzQX75fVeA7QNhmnZDfPPRcFWXgO0FEA5MITRffKUJjVQgxKCTWIlm27mn34CPEoIuzaemGYWzhEFQHbBqKeGQVx6gCIoX9xRbUsFAyvHoIMYox+J8llAwhhZGJaMNjUTa8DFGV540wQRjrXbO1rPK2j80rqsJDT+6fLS198vS/LLg67+b2DQ6OQanpky46Xmr6/q17e2c1GKjwiSiu4nFtGgQRnuC1u95O/nfDYSeuKIiCQYphWmexar4TectWxHDvjjfdd6HJCQRDz1VRBYQavxdVQ9GroLEqhBgUrYo1CoMgtiguEqJIjHUlDBIjlwceYWRdyvspaqyzZeFmdz/y6ZzBqiCA8opjVAXRlVdUA08IhRc4thVpCHlHrACIiSK8Y5ue/1Opwg3sA2O5yEBFEMaIQAQY+8sLF2WbdDGTpmEtHsrjz4znaeE0PT7+afGmVmaJz1v+nFvI1QwGGkn5TSx43QSEo5apXtoGeNAwOhkXCCvGiAkrKpAyvhjLnCce5LLnyveJkNu68B0if7NJ9FwVdeDZzyQC342spTGaQmNVCDFIWi8wgtiihdZLKxJrXSgw8sY77yen9hafJNcm9EDscpu7ZKs7d7wZTSQbYyzyA9lGGXTCKtPeNTwfeM0QchiteNOKFhvGqPRnXDGIOWfEW6w3EM8LHoUiQqXX0xAmynHzzpvrSl5RHQ9IGRBMx6uZzSziSulpctrqLOTKZ/menHnpL2Y9aTRCcfa/PLhy8CFsLDOmBgFFExg7fhhxKGSR7wNjltcZR4zLmBxRvjts30R+HufAtWrKk6HnqmgCvsNMaDDJwfepDTRWhRCDpnWxhlcsyzuWJ9bMK0cbZOn+K+/a4h50Z132QPBBOAxtzqKHXB+It68LM+y0sjP8seAFwPuFaLH1qcjhwavgi7gsLI/Oz7HhXDFaEVS8nzaQQxR5wXwwYK30ehr2wb6KQhvr5BXVYdeB15MFt2ycDY+0RoL6567Y6MTb3CXb3GwyoT94y2j8zWu8hwHDtjPrpc2UjqaxT/bNMboKYrsNL3ER5rllXPsUiXbGE3mPjGfGCyIu63vBuGzSG8ZkB+fWxPpveq6KJuH5ynO/znInWWisCiEGTetizQQZLe0hyxJreOG6sCg2ngAecCdduHIoYtSzGt6Ok3szg3UM56LiGHXBgL1t22vJ+be9kPxgze/cD2/Z8Jas9ck4d/MK4GnDUM7yhpEPQc5QLAhDBGBa3GHcIgSK1vbC2CavqK2Z4RiocrZp5wEXInnmD46HLZZtfBaPHfsalipk3DfuX1o4tYUJxCzDEuFeVGYfUcf5EgLGuTO+/YXbbUzmeXyrYNeqTlEhPVdFGzDW+d4Q3tzUb5TGaja2LFMTNtrU1NSsndi1dXXHDd9mD62PXAbbT5eW3xpWWhdrMDk5+YmbzxeTAcHaa7zGl50BwWtsaw8B/g2FT/YLQrd4uH3uig3BB+AwNfKS6Ms3r6n2pYkpjlEFjEoKLCAC8RZgdC6ZFlxVwq3wRrCPkKcBQxZBZFh/QmIQbwQl1cuAN9DPDcJYQPDFhFI2lVfUFB9+9HGy88XXksmNz7gqkpdNPuq+C/MW3evEGI2/eY33rl39uNt2+/OH3WeHEcQHIqQoxLYuFk6YV0UUIVam0AwFSBhnTEAw5hB7d26Py6Wsgl2r9ORELHquirbguUuUAx5gf/KiKhqr2fh2XV07DbuQ/TBJLwaLL9Zo3Oeq2D4k1urTF7EG3Cx/ABQ1vryDnGHZtueQe7BRHKEfFfPaboSu2SLEG3YcFy0xMINfJiywCIxRqiwyA0roCv/6pcqZIcUYLOtVIywM4ZcFBrAfXmaeQjsu8GOfDqOMASPc8oPYH+KLfhXB8bi2TRgWoh6EFHIP61RfzINxERtGjFAr8q6lYZ98j6goyXd++bRYayJfLQShkBjEZSdv9FwV/YAJC35D+LcqGqv5+KkqTMBXpan9iGZIizUar1XBPi+xVp++iTXghlsOW1Yzz1sd12sTfGfpzHpQVLMLPfiGsZFbRJ/KzKwxi44B24THAUGCQGJ/iCS8V1miiNnRPO9DGsQfxmOeEYwHy88tM1Hl5/Ug9mJEVgj2T34dxVcQjkUGOeTlu4n+g4cUj2tTYVQ+jGfGfVaOWRrEWhVjk4IpeNUYW0wEUGGU71rTfWL/ZfoDeq6KfsFEBb8JfBfKjFFDY7UYS1fBbquK76EbtN0njos17ilRb/Z3lXtj91VirT59FWs+DIh0G2TIo887R4+5BxoV8rq88GXZxsyaVeijhHsReNIw9mLDsULgHUPEYHiyL6osxuTRUJYZD1fszD37Lyp8wLmwT/+HGwMWY9bCLjE+89ZWy4P9sv//e89LUYZxW3lFoh6I5yqLqOfBdyAr7DYLcieLJiDSmFfaJkH4LOMZbx5jM6aaahmY6Iid3NBzVfQbnsNMyPH7UOY5q7EaRxO5Zpb2MuglmsrA+Q6ynkKbYIvbPcUmr3N/bD8Sa/UZmFjrMramyumXrA4+8Ia5UdGKvhWtuWKV34qKY4RAtBA6yY8kBiLGb5WS9HgiYpYI4FixIWOcU1rU0Vfy2R7e80cnKKsa6YTRfWNyf3JZZHESvIxlvIeiP3D/EWtNeTwtjLiKKMdDW8a7xjln5XsyzlmbkLHOmoN8t8uG+6bhWnGOTMIUoeeqGBRMljCJETvxqLEaB94WM8ir5DZZ3QIafw8DCBjOdxzEGviCvOw9ss9JrNVHYi3A/GXr3cOMsuWhh90wNwuDIMQjC2YjY4tj+FDNEHHFjyLGLqKoTugVxuXXJ/flGrkYi4jK2B9htuPc0uBF/NfbX0hu3Fyt1DliFGF6+J2Z0JsicUrf2J5/Rfdg3OIJIyyyDlaMo2qlT8YR4iomjIuwL0KMYwQYXjvLGUXg1anuyPG4Vnn5ojDuz1UxWBjjTJowSVY0IaexGo+th4sHpiyWFlMnjLLfmJgZF7EGFg7JPS4TDmn7kVirj8RaCkqY25pT5960K/iwG+ZGSAdliOnfW+9+WlDwIxZbHAMIV2S2HmPNjFteawpm7PM8HHgdmNmPBaMX4zl0jv8xbej++8/2lS6kwvYYvRjAwDkRWpkH4Wgx3ggxOCg0gviJWUQ9BOOCz1P0ow5MLuAZKILvbcx2PoRlssacfX/5u2xhH2BCBUPYvgNpxv25KroBY5vvE+HuWRNlGqvlqOodq+uVGxQID855nMRa1XBI24/EWn0k1lJs2X3QPcRYFDj0oBuFRmgHfVz3xL5er4+DiCgqjoHgwWvGjx7GaNN5MD55eV2cBwZi2RBLvH9pj4mtrYZ3AM9YrMeL7bgGfmgl146wzCxvH32hT3VD0ET7MK65V2U9T4wLPGJF3qYYzGOW510zr0GMBy4LvkdMjDDxwGRN2e8V58B5hiY7xv25KroDz2dChRmrockFjdXy4Bkra8hv37591piPqVfANog68/JYo8gJ3r2YfRi2LyuQkt4XgsXHBFpRKxJw7Jf9p4+LEOKzscX17BqYCOIz/O3vt2plzZBYgyqi3LaPFWuMCcaQjSdr/J/XeT+GKtfHXrfrzxgJ3Sv2HToPu7f+uXNfOe/0eKpCLbFGQikn3nbr56zLnZt2uYfYnEUPBx9yo9A+f+VG18cV63f2ej2DeciyQhcx3qishTFH1UNyceoYh7Hww0qBhDR4EUIhjUUQkkY/ffy11fB4xVS54zqR+xMKlUOoZeXRYRDXDa8T/YN7iRCK9bgyLvCsNuk5xWvG9zOLsrlteTCJQD4bYxvByXFjJy94JvCZ9Pbj/FwV3YTCOwi29LNYY7U82GhmoMaIDTAjmH+LwNC2/ee1GFvRF4l5DcPbiD0+9moIDH/eC30m3TDwi8SN7YvtuN5pQWHvVSFLrAHCg9djwyFjz4VjpgVaVmO7IrFY5frY65xLzBhhn3YNGCuhbfwWKzSzqCXW/JvaZsv6ArQBiwDzEJtYPBV8yI1C+8LVm10fr7p7a6/Xx9eYSocH4gVCLGGAIT4QSFXCpOpg3jV/zShe43yrriOFIWreQIxrBKjv6cKzEBKIBrOzCMW8hYe5XukQuhgviegeFipYNPYZF0xk+MtBNAHjJssbi3egbNXIWPiO2AQN3wcM3KLjIO7wUvuTPuP6XBXdht87Jlau2nB49rutsVoexIjZazFGaZntfSGILYih7gsF/m8igpYn2HwjPGtfZujTQiKD1+zzMeDUsBBCGoZ9+rhcD87NFyy+WExj58i5mFDgb/Zj+PsvQ55YY5/WF655Ebaf0HU0/HvCvrl/nIMP15B9+Ncxb9xUuT62XxtvHCv9Ge6bL/x437bndf++8i/n6J9zul9lkGctxcKVU+4hNnfJ1uBDbhTa3CXbXB8X3LLR9RmPGcaYLcyMkUX4Ft4lhAUJ2VVFUVMgEvEuGAhI//9lwaAkfBNCa6shpuh/VpEVjHGM8jzDNWREV8krEt2A8cKYyLvnCBtEfBvCKavSI5MCeLTahGcCnmcMW7yMnEdebirXgbFu12Ecn6tiOGCMUpWXCUmiLjRWq2FGLP8WgZFrBmyeqMD4te34TB6+0R8yin2BmCeEALuT7TC009i5s00R9M0EGPuKWd7AxAUtS5DY+dm2ecKlLHliDfx7wt952HZZ947rYdv4nqoseN8XS76Q8qlyfWyftLx7xTmYALN/887d72PRuMujllgbRb59/Tr3EDt76VPBh9wotLOX7nB9ZGFMK47x25ffd7PmiBD+j3ihMEIbRmcVEE8IR/JizNOWVyWyCAxN9mGiLLS2GqFceFPSeUeEt2G0+p6DLPzwNM4d8Sav2nDCdyHPa0Y4FR7bmHFRhVCuY164bVtg0CJcOReuR+g5wf/xVjDRA+P2XBXDB1EQ/PZ948aZaokaq+XwxVKWEW2YgCkyXm27GGEEZqSHtjfDnX0WYYKE/aSN9jJizRelZbwqJkiyztX6SYvxcJWhSKxBbDik7SdLrFk/ivbjw9gykZQ1fqpcH9ueViRCzZtmrWi82/WKGXtZSKylOOdH97iH2Lk3PRt8yI1Co8IVffzyotXTBteLyY83vuxmyxEtzJ6HQq26AHk0iB9mQfMqRMaCMck+6XuWKLWKfpaIzvUpU4AEr6UJtCbzisRgQIghjtIeLgrMME5ix0VVGPe2Nh9jlrFVt9pkVRjTjGdCHuk7XmgqaBpcK8Qr35lxeq7SVzGcMCFy3k3POK+axmo5MLbNeM0yzsH3NOQZxbHb+ZjIoqWNfzPwqxbeMMqItbJi0/D7Eeq7L0Zir00sMWKNa2vXM69vtp/QePA9nXnjJYQvlkIir8r1se1jBJV/jWK8Zf5adVWRWEvBA4wWesCNUqOP59382+Qrt72QLH/stVpeqn6Bcfrdew8kX7t93yeMwqowk3rRfQcK15PDI8aM6yMvvOOM0thCEwZhcSsff8N56bJEoRgeEGSMAxPeCHnGR9lxUQU8whyLc+D4TAB0AUKobY1FJkEIy2Ssc75Mhpyx8P6xea5+/spfBd9TU+tSM1unSWK8V7EerioGri8YfSPdF35lPFwhYsWaL0aqhCnaZ0NCxhcjTRMj1sAXlFkC2N4P9aGMJzaNf46h+1nl+tj2MWlX/jiLubex1zSP5u90g3AD6WTaDd0mY+VZ+/FU8rXJfU4AMQOOl6mrXjXj8gcPJv/58xd7/6sHnoHzp8Xqi0eKy7Kv3fWWE7bPZJTjz4N8v39Z8cK0ATsYD4hoHu4pognBxL9ly9zXAc/a5BOvuxybfh43Br5TTIIgIhFuhEE+ffC95Lxbfpuc+sN75K0QnQZv8LyfPJmc85Mdybxlo7fGmrW2xqpvlGbZbeaRKTKKTRSVDR2z4/sCoQlj2YgVa/4x+bssFgoZOo6JEbZpmjLXyg+HDAku209IrNl1jDlOGl8shfZd5frk7S+EbR9zb+uOBag1crk5dCy2c2Wxm1n0pWiSccutIM8Gj8+zh9+vtcZSP8D7x/khLrPWMCsDHpH/uGu/q/SXB8fFO7D0kVeic9V8ONdv3rFPhUVGjMcPvJucP/19enhPf+8rXrV/W7kvueLhbnjVssDTiFhDtP3LLbumjd89zggOPZNGoSlnbbhhAoaw4n9b9pRbNFo5a9Ww0L9QeJjvjSnyppj9V7X5dmkZAVJErF1a10A3wRE6Tt57dSlzrYrCIW0/IY3g398q5O27yvXJ218I2z7m3tYdC1Br5Ja5qVBW3NnNbGNAZjEOlaC+eN3jro/zl613faZym1W5w7Nmaywh4hAy/S7VnwWFHcgVIsSqiaIK7O+uJ4/k7ou+cx1MaCFo/Sp3RbAdBsCv9//JGa1d91yKOBDsjIvrpsdCmRzGJuDYeMSX1KiG2k8+/OgfybdXPJ3Mu2WP87CFnkmj0NLPVTE84CHn+cy/42gDNInZbRjxacwTE+PxsP1Ubb6dWdZWzSPWLq1roOcJjrz36lL2WvnbZy007d8Lw7+/Vcjbd5Xrk7e/ELZ9zL2tOxagr2Kt7PaxX4omuXb1zENslNdYmUitsYKgIK8qXeWONZas6putsTQoLOzMxA4Cq453DYPX9oexHVqagG0QsVbQAbhWhHhZlbsi/LyirNLrYriwMWBLPzAWGCdlPa5VYQxdP3XYfS+7nGtK1Ui+O5znN1f+dlqo7UnmXvNY8Jk0Ci39XBXdh7Bdnsv+b8A42gBNwqS82Xl+Pk/ZPJ8m7b+ytmcesedV10C3MMhQNcMqYiSWKtfKCn6kwyFtPyEBZNexzHEMfyyF8uWqXB/bX+hcQ9j2Mfe27liAWiO37E0tu32TX9ZY7tw0E8s9Z9HDwYfcKLTPX7nR9XHF+p29Xh/3FFCWPg3v2RpL5MmwRlk/PQmAN8sPV0So8QMb6+FK46+tRiioL8iA/SJQQwtjI/C4VlyTPDAEuF62+DaGtbxrww/jxveu8i//z1tEvSkYOybSEG3p9QEHDQVF+J7imWfsc457X/sg+frKvcmZl64du+eq6C58h5j043vrT7SMqw3QJGYs+0LDCkpg0GNsF9FEBT2jCWPZiLVLfdGaVYAji9icrDZs47J2OnC+Fv7qn5O9FuqDX2CkbF0KP5w2dD+rXB/bn8TaNGW3H4RYe/TZ37uH2GkXrwo+5EahnbFwjevj2m3P93o9AxUW/Sp3IZgtt6pvGKj9WIuN/DnOC/Hjgzcj71zz4LPmKbQ11/x+IN7YJn1Mw/LY8sqmpxfyBmZxY71yontQOZRS9WkvGv/n9aLKonVh7DCGwIRbyCvcT6yoCFUg8VYTVm05rxZGvHTjS2P7XBXdYyaPeH8wj3icbYCm8A1xE2bmKYopdQ5+BccmiszZvsoKpzRl7NKQgImhSMhUESOxlLXTDf9zdo3tPEMCyBezsQLJsIqioVBbqHJ9yp6LbR8jvvxrE7N9iHJ3I0XZm1p2+0GItfc++GtyykUr3YNs3s27gw+6YW6EIpE8Tf/eevfTRUQQY/yIFRUYwUBDKGGgIlqYQUf0tAEhZyEvFudYxbtmQsv/HMewsEp+wDEwizxgeMwQkSFjmc9yjPR7ful1MVwUrbHH63iT0ouoNwX7T4+d0IRAv2BJC4QZ50QYNWvN+ZMb/G1hxOP+XBXdgGc+45Fns0U8pNFYrQ8CzQpPIDx8w7yMsZoXChiC4/IZjpnG8uViqkuaUMT2TIulMnap7x2M7bfvpco61ypiJJaydrpPOhzSzjNLANk9ifW2gi/is4R/letj+5RYm6bs9oMQa0DSLQ8y4rpDD7thbnOXPOb6llcFCtHCjH3sulGsscQPIMLFX2OpCfB+5QkyRBYFUcpAqGfaA4LwxOjl3OlH7Dputn3agEe8mgckDdeKJoYHvhMYeEXfCd5HvNgi6k1Cjlx63CCIGH9MsvQDvheMbb6TTGgQEh2apOH7mg4jHvfnqhgsjFMmD/iNKiqapbFaH/N+YJCbpyhGKPnErOVlmFBjW/5N49ufRd49X0SkMbs0y6uTxs6J7Ys8hPTBrhsty7CvIkZiKWun+/hCk37beWYJIAQd18W2LxJsbO9fz6ztq1wfts871zS2fdY98vGvacz2ISTWAhAawMOMUIHQw26Y21mXPeD6Rlx+HuSd5HkRQlg4FAIKAxJBVHfxanIK8kIdmd3nWP6Mfh4YkSGjmzC2b0zuS771s32lDV8MVs7TQuP8vKIQ5iFpYmFv0T7mQWWsxWCLZDOJ0RSMJcZUyMhksoLvXFswrvkOcgzOgTDoou8IojIdRqznqhgUfCf5DjPREDORqLFaH9/eKzLa8/DFC/tBwPlGOgKI/ZrRnyeKzPPj78tgn/zfzpUWEoh+iCLnZufCvyFDnHOxc7PPpLdjG45lQocW8g4ado782zRl7fQ0/uet5d13/3pyndg2ff/YJ/cu5h5Dletj5xA7Rm370D1P41+TmO1DSKwFIDSAhxmhAoQMhB54w9n2JifPv9317fCbxaFaGGWh/JwYbI0lfiAJkULExQoqI7ZEPx6xUO5BCDx19CkN5/uvt7+Q3N6r8FcWvB7X/Gqm6ASGKtcuD7x7WZ430R0YF4h7xmIZEDd8rsxkRx6MFQzNEIw5JlaaXhsRA5cKsV+f3Oe+Y3gXY77DFkacFpZ6ropBwPeG36Ey3m6N1WbwxQcNz0gVzBYsanhd8ox48AVbXssy2hFl5t0JtRB8xgREUUOI+CIyRBUxEktZOz1E+hoXCSDuWXqsZLWYe1zl+tj+JdamKbv9oMQafGfpg+6BNkphEBb+cP4V9/V6mQ9GIAKkTpU79oGhS+gJs/IIuLRXK4QZoJZHlgd5YfwYF+WYAdXz0vlvGNTkGv38qTddxcsqcL708eYtr34qryhEkfdNDB7uIaKjbJitgSCvsoh6GsY3YyVvfJvnqy58N/GIIzQ5d4RXUciYT1EYsZ6rol8wbnkm4+Gtkk+tsVofX2TVteMQeogAM8StYeRbqGUsGPt4uNKii/9zjCJRifji877HzM4jD2zg0HH9PpinLo8qYiSWsnZ6CPrgi69YAUT/uQ5p4cb14roViVijyvWxY0msTVN2+0GKtU07D7iH2infuyM5b/lzwQffcLW9yWcX3OX6tO6Jfb1eFoOhiXeriSp3GHBmCGJYYmBmzdTzHj+ysSAoszwPBsYuQso3eukfRql9lnOrGr7Gvr55x75oo5ljyrvWTbiXjL+6lTvTZf6rEOM5LjO5kQajFkGKxxmhRZ+rfAcIE0VU5oVI6rkq+gFjkGc5UQ5Vv3saq0KIrtCYWENYFTWUcZnt25w9iOGCa3/pHmysSRJ++A1Pm7jqEdeXry6+f/rHq9xMPx4GjLimqtzx40nJe7x2lgPjG4cIOH5oy4R1WU5PnvcBj5q/LpV5w3zBFCo+Egvn8K2f7U/+8+cvRhnN5l0bdOl18UkYF+niGFVhLCP6qhaUKZOTGRs2DPSRMUofmcDge0GIcFXDFo8c5xmzcL6eq6JNmNhgTFeZuEijsSqE6AKNibU226DE2q4Dr7uH20kX3Z6cu2x4S/jOu3nPbJz69ucP93pXDsQUP4AxxlgZCE/Bw4Q4Y2YfQbVu9x9dnltZEF2hRb0N9u+fP2XHOY5voCK4OJcqRiv5PfTFwjJjCpVgWOA5Ed2ByQMEVlXhkgbPVdVwSvNAx4JYyzNSGZsIRyYJ6CMTMHkTHDFYGHHsZI6eq6IN8IYz6cB3oKnwco1VIUQXkFgrYOHKKfeAm3P5g8GH4DC0OVfMlCH+7o0P93pVDYQOgq3JKnc+7B/hcv5tLyQ3PPJy6aqM/EBTDCGUL2aFIswAR1Txo84PfBqM2LKi1ASaeUDwHHK8ooqPbM/n5F3rBlRBjVljryzmecpbRD0NBREIbSwjGhFqnL//Gb4P9IvxjqhiQqMpY9bCiPMmSULouSqahN8kvitM2MV4ocugsSqEGDS1xNo48Orbf55dIHPukq3BB2GX2xeve2L63Fe48z/w6h97vaoOni9ESEgQNQEi6qoNh1w5fAxLDEE8ErHGM16RUMgZoY0W3lhUqS8dLhlDKK+I88ZwDglCH7ZrojhEk+w5eMSVr17+0G+Sq+7e6n7kCZ858wd3uEaCOq/xHtvcv3Wv+8wwh9cUFceoCyGNTHbECnPEFVVUy8Jkw+Z9f3KfJdSYY2LENl0tEkGYDiOORc9V0RQ8z/EUl/FAl0FjVQgxaCTWIli1eY970J08fzI5e+mO4AOxi+3LN+5058y5T258pteb+lCAoIkqd2ksh8uvFomBiXAyg5M1r/JAgLGtL8QwKhFn7BdvRfoYaTgP9hHbP86RWd3QjC7CMR1qmYb3+HzTxnQZPvzo4+Tpfa8k16zalpzzo3vcmKnSJi65y+1j255Dbp/DQlkhVRUEFIKwaLIDDxlirSz0Ywne6env/5XT4+7xA+817mkwQmHEZdBzVdSBcc0Y5NnZ9vdWY1UIMUgk1iLBAOWBR2WoYYhdn3fz7tnKT4t+trnXi+agMEHdKndpEDZZs/Tk/ViI2oL7f5dbUjy9H0IayVfjBx2DHIO2CARibA5OXl4R14frhFGRB58fhHeNmdYr79rivGWMFWufXfBzFzozsXgq+cI1W5IvXf/k9A//M27NIRp/8xrvsQ3bnvrDT4o89nnZ5KOdn81FuCPgy4a+VmXts59cRD2NiffYkEnyPtknn+H7gVf6sunx3lRBoBBFfYhFz1VRBbzfjD+erU1PGmahsSqEGBQSa5EQ3kXYFw++0y6+p+MLZe5NTr/kPneuVLM69rfmw7r4gUQAFS3+HIt5xGJyaRBbiDG2R1SlPVKIOIxv2xfbPLDrbefRiA0rw3AnnKwItkvnCKXhWuGJxIjOwgz0fgkGQnsIYbTwHsJkTrt4lRNeCLHwuCpu5970rKs6dvolq90+Z/Z9m8v76OKCwYw7vK5thVBlwfglhDA0bmKEO2PKtmOs8z30vc58RxjvbXjVOG5TodB6roqy4HVmzBctZ9E0GqtCiEEhsVaC94996PJ2Zh7Wqzo5u8Zs2oyhfJsLZ3vnaLkiHWXAWCOvrEqVuzR4nkK5ZnkQrsix8bSZR8EMSFvDjG2+MS3qLrrvgPPMxYIRjbFbJB6Z3Y0pEW2iIE8slim9XhXGw7WrH58VaSdduCL53BUb3bgJjac6jX1S8tqqkHFMBCJCsQsgeLjejJV+w/hCaK18/PXeKzMgrvhOZYXEEsZrExV4FRgzWYIslEdZF47PsfPCiMui56qIge+MrdMZEx3RBhqrQohBILFWErwDFFfgQXjy/DuSs5c+FXxgDqKd85MdySnfv3P2Id0PT4aFkJWpcpcmr4pjLHgVMHwxJAnR3PGHP7vzWjX94/4fd814HspCbl5elbuy4squVZ6hESv+qrBl98HkjAUz+Qt4vahu1g9jg2PMWcSM9Iyn7dT5K5MNO17qndVgwPCrWhyjKRCLTDL4kwiIK87Lh+8H4xAjlfGRFwLsQ9gvEw4cpwnKhBGXRc9VkQe/DUQ6kCMZM/bbRGNVCNFvJNYq8N4Hf03mL5sphYsBOnH15uCDs5+N3CFLJP729ev6OpuGFwAjrmqSt61P1gQYplRzJOzw3+/Yl3zltueT/37goDPOy0I5aLwcIdhflaIgeCYwurM8dgg1PIVVzjcPkstnxuttyRkL19QKdazaOOaZl84sMkv76donp/vZn3yTNHlhiP2E3BvGA/edsYu44nuER5hxTKgxrzFxUGXJDCYumvhukRfHeVSpThmLnqsihD0zGceD/r4aGqtCiH4isVYRjMwb1mzvPaxvc0bouTftCj5E22x4Lc66/MHZ8yDMbBAGMHksGHMYdWWw2XqM06ZZsvGwu0bfmNznSpjj/Sv7Y4/oCwmymLyiLCj8gEclq8/MILP/JiBXwdYJckbF4qlPjaF+Nwwb87Jh8BBa1E8w+vBQNeVxqgteYb47y6cF5I/W/cEJLL4T5FqSw1jHQGVSAG9une9XyAPYFnquCh8K2fDd6Fcubxk0VoUQ/UJirSYPPbl/Nv/n5Ol/qYo37+b2E485BvlAJ110PB+Ita4GCaFaeIXKGMHk1eQV3qgKuWzfuvNFJ3z+d8NhJ37wUtgMbexaWuwnHSpHjhD7KVpGIA9yLzi3kCGOOMRrV3cW+a13P0i+dvWambE5fzL50vXbg2NpEI3QIUKIODdCivoVrtNkcYymYMLif9cfctflh2t+58R8k5MXjN+q3jXGIOO0qUJCsei5Ot4w/vF888wuOwHYbzRWhRBtI7HWABRMoDQuD0z30Jw2jP9p8aZWqkWdt/w5V23PDF0anpOulEfHGxAbXoYoYda06Yp15JJhkH/75y8lh6cNYf427xihZBQywdvAebJt3rmSH4GXwxeg5AwhMuuCB4UQ0BB47ep41/ComVCjpD5VGkPjaZCNGWGS9DlHBBuhRW3C7Dz3vcniGFVhXDGO8PARavvfv/x98oM1v3cerKY9fnjXGMNVBCrfZ8Z63YmDKui5Op4cfPuY+07wnB7EuKuCxqoQok0k1hqEh+XxkLOZSnvkBxH2VafaHp8lHv3MS38xO4tGo4zw/peb90rVwWbi01XuQiBImqgk6WMlyzGEmZWFUMgiApH8G15nezxdWUY8xqoJJ2Z8EX9V8/N8OAeuVcjrUaf0OiEwF9+6yY2RU394d6dLTHNulMG28dxW+I6F2w6qihxwL5kcICSXc8HjxSQCIor/40Hge1Nnoeks8IyVrbZKnhzjc9Dhonqujg/kafJd4HsyjGisCiHaQGKtBXYdeD1ZcMvG2dAIa3gRKJPOg3vukm0uFAyPBzNlNP7mNd7jwcy2M2ulHF+vin2yb47RVfAaFOW44OVoItTPB7GFkKJYAx4BfviBY+QVA+FzGKaII8QbIs4XSewPAxoQVk1WEMRQ59weP/Be75XjVC29vmzd026sMPPaRY9auuFhs1liFp5tGkJeubdtFsfIwyqV4tVjHHGv/fHlL1vBWMXjW7SIellMEMZ615hE4TtcxRvXFuP+XB1lmBDgmY2nOav40jChsSqEaBKJtRYhFG3TzgMuPOLMHxwPWSjb+Cyzdeyr7VCxprAy9VkzpPwo1wnzS2OiB4GG9wvD1M/74VgcMw8MZc4XY5lztwp8vM7/X+31qWljwq5V2uuDN6isd438iZlxs8L96IfEURfb2Ut3TAu2mUpmdz+yu9eb+jAG+lUcwwcvGcURODaNvMxQ7o0VAPFFEYYruZ98vkkQhDEikMkJJj2yPM2DZpyfq6MIz1jGO5Ngg/biNo3GqhCiCSTW+sSHH32c7HzxNVdCnQpSl00+6kIY5i261z2IafzNa7zHwsVsu/35w+6zwwjiA0M0XYgDQYRwasqrxg88IY+EMgKCjVlaH47FMTFEY8AbYwuwsm8KQPBvW4UW7FqlDWTy2mKN9jfeeT85tbcANbOyIVHU5TZ3yVZ37swcN1FwhHvez+IYjEO8ZnjPuJd404qK0GQtW4F4a9obiMeb88qbbLDw2zrFc/rJOD5XRwm+L4zJJifuuorGqhCiKq2KtUOHDiWf+cxnkomJid4rYtzA2MT4s+qLGNB4wGJFUxHsj3BBX5whqkKlnjlmkXctDfun5P+l6/7gBMXyx15rJF8tBIYL18b3spjnxfcSZnHlXVuc2Dnrsgc+JYSGpc1Z9JDrAzl3dUHo0riHbUJ4LV4BvLnkozEZEeMNtTy6rHvL+9z7JvPs8sJ4mSiwMGIh2oTvB5MZPO/aep4KIcSo0KpY279/vxNrF1xwQe8VMY7gGSIUDM8DM6hW+KMJ+MEnz8wMcjM4swz0quGXGLj//rOZohDsn/XX8OA1HbaDMY03yDf2OWbI++JDkjki56QLVw5FnlpWI5H+5J53sE5ORtvFMRDRLFXBWGBMkVuI56oMMTmJTBRwjKZCbxGGCMC0B5cJAr6jTRf8ESINE3c8Pxn/MZNQQggx7vTFsyaxJhAcVz58yBmeWYU+ykIekIlAg9BFC4cM8dyr5dcwM+/WL3a95cQhn8VzZ4sXx4S7lYHQON9TaMf3PW5pCJ1B4Hzuig1BETRMjZLX9OWb16zt9a4cbRXHwLBkDTREIN5iBGFVrwDfgdh8ROtPWTGYBZMnGMoG3x8/jFiItrDlM/qdQyqEEMNM6zlrJ5xwghNsR48qtGacQeDMv++AM1CbgJBB9uUb5BwDMVhUGAFPXBnvGsIJzxbHwtDwDWxeQzSyLhBJ8hjWdWeL6QfniOfGyCu9vm3PISduqKjYj8VY225URTvl+3e6Pm3Y8VKvl3EQghgzBmIxYc4YQJjzL94uXq8D9xfhF4t5CuseFxi/fHcQmuwvHUYsRNMwziwHuMmwXiGEGAdaF2tr1651Yu26667rvSLGEQTMt6aFzn/d81LtKnd4JTCcqSLmg1EdE2LJ5xFXMV4NDFrfA0KlyKyiD3jtzKjHk1jHKOF64U0xgx5RyH5D3qLvLH3QCRsWSg2Jn2FsFEgp611rsjgGY8uWc0AkWZXRJmCclvXumqjKWkS9LIRfkl/HJIAfRixE0/DMougOLfT8EkIIkU/rYg0Qagi2hQsXysM2puCZwtisu+YVHhNEC8IoDULJ1lYrAsMhZg0zDFp/O84bwZYH4WqE+SC2aPxdJYSN8Edmoq2veNbSlQ3fOXrMiRoWX+3y4tdlG941W/z11beLx4rlKtYpjsE94l6Tg8a+8Gw2lSvmw/7xDJeFcEVyfYryF2Ng8oHv0f+956XW8vqEsHBfxqwmBIQQohp9EWtAsRFy10y04XHjtZhG7psYXjCC/aIGeD74f1nPE7Oy/PCHDF28Hhifsd6PtMcsBIYGHhB/G/4uyh/zYR942Tg3xCT/L4N5i7h2dh19AWHrqp1+yeqg6BnmRlVL+la07hr3gvtUpTgG95PQSSYSuEfcq7L3qAy2bEVV6Cte4Spiz4dJh29M7k8ua7DYjxA+hIfz7MKTLIQQojqtijWEFuKsblOBkuEGjxBGsA8ekDK5Rcz+5y0UHFpbrQgMdAyKLLJy2ygoUjZBHhGJmKAPGNscN1bwYeDzGbZnhtq/lvOXrXeCZuLqzUHBM8zNQiEJ88yianGMg28fc15KxC9eVsRP2x4mPAuIyrql8c27XFVU8jk+f/idv7rzaVOcivGDZx0RCXiB3wksBC+EEKIcEmuiVRAYX5/cFwwns1DBohBBjFyEEyIpCwz2UGhkHnjXMNZD3jiKSGTlFeHtQnRVBc/irY/NeNuoKMnMc1GIEOKOPv7pL3+f9a4d+9vf3QLSCJpzb9oVFDzD3AjrZCkC+vfWu5++t1yzMsUxMBy5jhaeivDvpzGJ8GccNwFjnXEQO9lhmNCz7wrnhFEtRBPwTGViiQm6omeaEEKIOFoVa+Sn+eGMVZvCIIcXvEBZVQwBAVZU5Y6iCszUZm2DAYqXrgqcXygHiFC1PA8IQg7vTB3w5OARxFjm/DmPPPGAwEOcsB3/btl90AmZ0y5eFRQ7o9AI76SP657Y17sKx2HsFBXHIMwRrxneM0Kyml5mIRbOg3vcpBeL4jOMw1gPLdulw4i5dkVjXYgYeJYxgVA1H1kIIUSYvuWsifED7w9etTxjEmMRIZZV5Y4iD4iZvBA1QuD8Mvdl4BzT3jWMjaK8IrwyeSK0LAg/hATnwvUIlYfn/xQ3uXP7G87ovvGh3U7IzFn0cFDojEL7/JUbXR9XrN/ZuwozmIcsa1wgisx7yfUklNTPPew3FC5BMDYN457JjqK+cZ2ywogRanXy6MR4w9hi0ovxFYqgEEIIUQ+JNdEahKfFiCh+7DEW0x4uZmqLPAcIGDwWdYwE3/vH/jhmUUglHjCEVVpQ1QWjm9A0Qh4RZAhRKmgaZnQvfeTV5D8nZ8TaxOKpoNAZhfaFqze7Pl5199beFZgR01ybtBeSMcAYIgyL8YRAqlKFs2mYCOB8CRFrA75nhNNmwRhFKCJes+B6ySMiykJUA88jnqFZEydCCCHqIbEmWgHDFK9GKB8shIVoWUEPPEv8vygnh3wvPAt1wOjnXDmHMnlFGMBtho+x1pcVwcCjhncIw5vz/c9pgfovtz7nQiDnLtkaFDqj0OYu2ebE2oJbNrprYsUxbI09DETCARkDjBfWRmtLFFXFwlbbAoFP/7OKrBSFEQOTE1k5mkKEQNzzbCqzuLsQQojySKyJVsA4LLv4tQm8qeffcYZ3TGl/vApNGAt41iafeD36uICwa9MINzDGMYwQkRhHCJKnD76X/PPy55Kzl+6Ybk8Fhc4oNPqHWGNxbCuO8duX33ciHRHC/xkDobDRLsBkBfes7fAwJhoIC01/F2LCiA3Gl02WCJEF3zMmkRD3XZsYEUKIUaTvYo2CISySPTEx8YmKjyeeeKJbf2379u1aOHvIwfuB6KmSI7Rl2lg8/7YXkg3PFZfGxxDGWG8i/AZj999W7iuVV8RxOX4/Q+0QLIg1BMB5Nz0zLdieT867eXdQ6IxCo8olYu3Li1ZP9/nF5McbX3Zhr3iSCJON9dwOCiYBCBHrB4wNvncWwhsTRuzD95btq3xvxXhASDbin0mqrn/3hBBiVOibWKOioy2KXdROOOGEZPfu/IVwRXdhhp58obIgevAOkI8VY2RijGYVJikLhsfXbt+XLCnpKcMQr9LXunz40T+Sif+dSs675bmgyBmlhlg796Znk39Z8UKy/LHXhqaIAeMXMe/nHLYNXmGO+cgLM97pojDiNExWDGI8i+6DN5sxlbc2pRBCiObpi1hDqCHAfDGGcFu7du1sw9uGd80XbXjZxHCRtz5ZHnip8JZYoQ+8R/w/z2tWZW21LMgrunFa+JUNWcMb0c91qqgayTXiPM+/+Rm3Ftm5y0bfs3bO//xidl0+ChqwwHjXZ/Yp6GHjuZ+s3fVW8pXbXkieqZBPSVhbVa+4GE14lvN8ZFzEhogLIYRojr6INRNhiLSpqaneq2EIk/RDJPm/GB6oKlc27wVjgGp2fkU7XiPUJqvKHR6DqmurpcGjZyINoyR2kWWD6oNW8KINKCiCUEGkcCzO8YXX/5J8ZflvkzMvXTs2OWsGBiMeTctXQzB3DcYS5xcbgtgUHJfvxdLp701srloavnfyrglg/OJtpfV7LAshhJihdbGGd8yEWpnFrclf43MIPTEcUAQDQVHWq4b3AS9aejYfQxPvWajKXZ211dJwfNY4A7w1CLcyifOcB57AJrGiIlSBxOjHS2OixMJFv7PiSSdkRrka5Beve9z1cf6y9a7vPtwrPGyMOa4HgrYLpfqB8FxEdT+xcWFCC0GL8Cr7fWTs8x1QTtJ4w6QIwp/nW9kxJIQQojlaF2vmJSPUsQwUGbHQyTIiTwwGfswJfyxbyt5C27IMQ2Zz8Sb5Ve44Vt211QzLK/JnjTF2y1R55DwIEWrCoMFAssWcmc1+/MB7nxCx/G3hoteunhEyo7zO2kRgnbUQB948vqg43ljyawaFVTXtp9hJhxED45H80SoTCQi9fotN0R14LvNMG+T3SAghxAytizXEFq1KhUfy2KoIPdF/CH0sm7vFumEYBEUFGKxku+WnNbG2moFRmjZmEUScF/lhsXA+VQ0b+o9hjNhFuJLAn17wGTC+LVyUv+/cNJPPNWfRw0GhMwrt81dudH1csX5n7yrkg2ih8AxjEZHPtex3+BZCH2O3X/jjIg2CkTHFNSkDExDyro0f3G/GL9+ffhbGEUIIkU2rYo18M8QW3rUqWAglIZGiu5i4KZM7hAcJYzBWECHUEGzkhjW1tlqeQUp4Xezi2MD5lMl1Q1TYAtycA+sWFV2LdLjoo8/+3gkZFsYOCZ1RaGcsXOP6uHbb867PZeB62qLiGKD9WIuN7wDfBd8b2jZZYcSG5bHR/zKEJjLE6IJHmAkjvjNtf0+EEELE0xexRuXHKtT9vOgPhA2WETZ4yjCgy3qiEDf/+fMXk29M7qtUOCEN4ior1AtjBcMlVoByPjGhb4hO8pm+Pt0HBARhozGGPdcYD4mfk/XeB39NTrlopRMz80ZwrTUqXZ504Uz/3nq3ehERri9jh/xHRAv3POS5bAK+B2W9WHWwcVE07ggTRUSWycXkGjGmVVhi9OH7wTOZPFkhhBDdQmJN1AIjkR/5WA9ZKAetDFdvOJx862f1F8LGaMV4zduPeb5iQfyFDHXEKQVREAqEF2FglymEQbgonw2FJVF4AzFDbldI8Axzm7vkMdc3vxJkXfDM4oni3lO8hWvblBeByYcqy1ZUJTaM2LDty4gvrpWfBydGC55/eFAp0lN2TT4hhBD9oS9irWpFRxbGlljrNngpYotxYBhQ2r9O4QK8I1c8fMgds45RjKGOaMrDvGuxRVP8XDrEGKGUnC8GMuFkVcr7F4WLEh6IoCFcMCR4hrmdddkDrm/k5jUN3ja8CIhx7g9ium6OTpVlK6pSNozYIIeP84yd7EDY4V1T/tLoQXgsYwGxVnfySwghRHu0KtYAsUU7cqR87sPk5KT7rAqMdBMECQZjjAhB+CCQMAyqYmurYWgjiqrO+BPaiHcvJvwQbwQGTQwffvQPdz2s3D6eNgRcVVFJfxESeblGhAciaAgXJGwwJHqGs+1NTp5/u+vb4TfbFUBcZ8Q015oKnIi4mLHhU2ac1IXzZZxVLWhD1cxrfhU/2UHp9jrfW9E9GK+MoX6G7AohhKhG62KtidL9eNhE90AsxRpxbIchXMcb5q+tZmtK4b0qC96UMh4QjPA87xohlVwLjB/E5DW/OlyYQ1QEHg28ejHhot9Z+qATNaMUCmkhkOdfcV+vl+3D2MSIRWxzLxFwMaFhfK6MB7YOdcOIgfOlj7a2YBEW6ownRgw33HueVYyhMvmLQgghBkfrYq3qothWtl+LYncTjEaKZMQYcIQ9lgm9CoGRgRDyjxfjeUqDQV02r4jPIAz9z9B/yrPTLwwfRCTnhpeR86wD14ncNhOmRWzaecAJm1O+d0dy3vLnguJnuNre5LML7nJ9WvfEvl4v+wthf5ZnaOI+y9vGexb+2iY2LuqEERvsizyl2CUGOKa8a8MNBWMIyyaEvO5kkhBCiP7RulgD864h2BBveRAuSal+th81r9qeg0dcjtHyh37jFvn97o0PJ19dfH9y5g/ucA0vAq/xHtvcv3Wv+8zH/+hePgGGGyWei8CQxeBF3NTBzwfzIXeHkMPYWeIiL1kWHHvzvj+5EDlCyDgm1yBULRKD2taEKwuCkP1TMbIMF1z7SyduWJcsLICGp01c9YjrC9+NQY997geTAdwTvEuMeT/sFwHH+C6zbEUVOI+6YcRpMN4595jvg3nXuuiNGaXnalvw/GRii5xFIYQQw0VfxBoeNQtpNG8Z+WiERvqNQiK2DY1thpkPP/o4eXrfK8k1q7Yl5/zoHmeAVmkTl9zl9rFtzyG3z0GDBwmvWpEAw0DAwIsJJSsib201BBSGSNH5PH7gvUp5RQjCJZteTs6fFhJXrj/k9pOX00RoZlWj+tbHqoWL7jrwuhsrJ110e3LusuEt4z/v5j2zuWrbnz/c6103QNzgYULg4KEg32fd7j+6+9U2VcdFEVYVNaZQCQV5YosJtcmoPlfbgPFiY7btCQUhhBDt0BexBgg287AVNYTd1NRU75PDx4FX/5hcedcWN6vrGwefXfDzZM4V65OJxVPJF67Zknzp+ieTL9/4jCsMQeNvXuM9tmHbU3/4SWOEfV42+ag7xqBAOBEilgdGIN4nhE5dmNVnX3lhlGufza9yh9FC+GOsxwvDnH3yGUIgmZG+bNpAj8kVssIreecbguMRmlb2c8bClVNujMy5/MGgEBqGxpinD3hCugwTEQiX8297IbnhkZdLV2UsQxNhxHngOcSYL6r4yAQFwm5Q3rVRf642Dc8hvLGI/LqRDUIIIQZH38SaQRgkYY6+p80aYg4PW5XKkV3g1bf/7EJtbKHi//PdFclpF69yBgIGQ8g4jWnn3vSsCw07/ZLVbp9mYGCct10pL42JsLycB4w+jLqmFljFgxETFog3C+Mk5H1AZBWtmYYxTNgm21nIG4sJGwhP+hVTKRBDvkwRE7x1GMx1jCrGn429uUu2BsdSl9sXr3tidnwPg9GMiLpqwyEn5MlbJPwVr2qT+UBNhREXwXkzKVEkCNmu6HvUNOPwXG0anluMGybVmvbGCiGE6C99F2ujyDtHjyXXrn581pg46cIVyeeu2JjMu7n5cDT2SV6ShYpxTAwZDJp+QO4ORmoWGKoYfbGFC2Ig5CzGI4ZRgiGZrnJnHoEsLx/7RughQhFZVATMEmS8X7Q+G5AHFGvUcnyO3US46KrNe9y4OHn+ZHL20h3BMdTF9uUbd7pz5twnNz7T6013sRwu/54RZobXmXvJePKFfhUYF02FEcdAlcCiUEvzUPcjpG6cnqtNwvOJcVN1aQchhBDdQmKtJlt2H0zOWDBjZDI7SwhaP3KGOMacRQ+7Y3LsU+evTDbsqFeFsAgMtDzPEoYchThiCo/EgqHKDHEseAbSYjGUa0PeHdUW2TchZmxD2FARluNT5IHgWmAwFYWWUawC475J45c8HGdwfu+Oochfw1C26o+Lfra514tuk7dsBeOI8cc4JKw1dmz5NBlGHAtjlu8J+XF5mPe5TcbpudoUPJNueOQV5+HVIuZCCDE6dFqsEQ65f/9+t+ZaF8EDMGNM3JacsXBNrZCcqo1jnnnpTCVA2k/XPjltdLWT24KBludVwlCg5c3Ml4W1rmJL2BsYKogwvFuW74bxy9+EVOKpQ3Cx75gFvdPQxzzvooFBTx5aFoS2cR5NhYsa3H9yvhgPp118T8cXy96bnH7Jfe5cqWh57G/dNzK5b4ypmGUrEFvmtcXrFiPKyZdsY1zEgMGPsZ9XNdC8a215bsbtudoEPN+4J0QVNPn8FUIIMXhaFWsILXLRqPJYBfLX+Dz/dgkMSivmwAwsuROhH/x+NhZEttng+cvWJ+8fa3YBW4oQYAxkedUQPnjVijxOZcDowGiNMYrTEILGZ5dPG8pXPnzICSwzmDEy6xg0nA9es6LcJDOgQpgHsK1S2tx/ypczHsjv6aKHDY/aTL7Qba6qH2FvwwCeJ4R4GRgr5HvhabP7HspDY7umw4jLwnkx2ZEnFgkVrlJZNY9xfK42AZ7OQYl7IYQQ7dNpsWafpyBJV3jr3Q+Sr129xv14k2Pzpeu3B3/kB9HOXvrU9DnNVEpjbaEmk+QxzLIKZuBtw8BsuggCAjG0tloMCCXyb7guP1zzO1dgpEhclQFvSYx3jeuWDmVDKIZy65qG+884mBmrd7jxkR4zg2rn/GRHcsr373TnhlAbloIOsctW5MFEAveeyQMmEcxDxbhoOoy4KoQfMyGRF4bJ2K6yZmGIcX2u1oGJMyYOmACoEiEghBBiOBgKsVb1803DzK8ZFJR+pppY6Md9kA0PCp4UMyze+6B+mW1mbBFjIW8UM+zMwrdRBAFDNqZUvkFeEMIRI5LqfP/9y98n/+/a37tzb9LjBxjtGNtFRjseknQOEN49coPqePdi4f7jEWA8OG/F1ZuD46afjRLqVkzk29evGxqPGlCVNEakx8CYJCyXsEPG6/enje4f/6o/4yIGCpzw3c7ybCPUEAp1z3dcn6t14J7wnGM8Nv1sE0II0S06LdYo898VsUa+wsW3buoZFHd3Og+IcyNXiXMld6lOrgWGGKF8oRl0Zt0Jv6lb9S4EwgsxVGSIMLuMYKRKJdvj8SIviLwfE1N4MdpYUBgPSFE4HMfnPCx8lFLa5Mz108Di/t+wZrsbDzRycc69aVdw7LTZMHjPuvzB2fOg2l6X84DSWNGPJj20xrJpo/vbP38x+dbP9ruxjFe5C6KNyRImO7L6jCewzBIVacb1uVoHnsU8dxH6QgghRp/OirXdu3fPrsV23XXX9V4dHMvWPe1+pAmH6eLMb7phGFvoDtUBq4IhxgxuGqvS2FQYVBrye/LWVrNQMkK1EGKsU+bn0yHayKMDjF7WXyuqclcWE2JF3jWMb7yTGFdthIvG8tCT+2fLoJ88/S+LA8+7uX3jmGNQFv2ki46XRb9/697eWQ0PeEPbyDG0Nc4YF4xhvnMIer5fePEGXdmPCQZEWUg8MjHCZE5VYTmuz9UqcI15puGFbXMRdiGEEN2iUbFm4qzpxn4HCUYuP86EkXUp76eosc6WhZvd/cjuXm/iwXBkBjddwQ6jEgOtzZldQsPSa6vhLaO6IoYtDcOZ19KECoDgySJkK686YxXwrBWJQDx//++04G0rXLQMrBtFefyZ8TwtnKbHxz8t3tSKR+O85c+5RYfNuKVRQGIYFrxOw3eA74I/IdAEeWHE5CExvhjLTDaw7aC8bYQkZ02ekH9Zxbs2rs/VKvCcQzAz8dOGZ1cIIUR3aVSsHTp0yHnRzCPWRCMUcpC88c77yam9hVLJtQn9eHe5zV2y1Z073oyyifHkf6XXU0L0MOvPbHtbYKRiwALHw2uG9wyjFW9aUdhlVml9RGbTVdMI1+S8svJ6YN8bf0nOn74Xv/lDd5agQDAdr7w3s+AwZdLJaauz6DCf5Xty5qW/mPWk0Qgb2/9yO5Uv+wHfAzxgTRIbRoxAZMxyDmzPd6/fop9zQCyEvlfWjzJCdpyfq2Wx3MG2KscKIYToNq2FQbI22tTUlBNcExMTzjtWtnVhfbUr79rifpTPuuyB4I/2MLQ5ix5yfSA3JBZmbxEhviHJrD4zuxTIaBNCfZZteXV2fSqOiVchxhgkrwjDMSsnjPfpV16Vu7JgwHKuIRBxGFr/u+FQJ42tXQdeTxbcsnE2PNIaxRQ+d8VGJ97mLtnmPB+EqeEto/E3r/EexjbbzqyXNlPmnMY+2TfHGGao1lgn1C9E1TBiPsf3gzGOeEPENe3ty8I86kyepCFENG8NxjTj+lwtC88WxknM+nxCCCFGk04XGBk0eAL4MT7pwpVDkU+R1fB2nNybxY41nDESMMB88GphILYVioWwWfWbI8lXbnsh+f7q3znjD89VGQgXKzIaKd6AAZTnDSuDCdu0twPjlnBNPDKIXsIwuwoV+TbtPOBCJM/8wfGwxbKNz+KxY1+DrpjXFORs1imikaaJMGK+g0xgMN4Zewi4fpRvZ4yHJjtskiRGOI7zczUWnnvcW563jBchhBDji8RaDoRu8UP8uSs2BH+sh6mRl0RfvnlN8QLjFtqHAWbgFUJ4NF3FEKFDxTlCrDD2rtt02JXbrwKzzyTfxxiMVtShrBjMglw4X9xyneiTFTkBjjcMhQE+/OjjZOeLryWTG59xVSQvm3zUfRfmLbrXiTEaf/Ma7127+nG37fbnD7vPjhIIIsRaUxMUbYQRU4CE/TEBwb4Rlm162xBqockJwo/5HhQxrs/VWHhG8BzjnrY1MSaEEGJ4aFWsEcaIYCOXbdjYtueQ+xGmOEI/Kua13Qhds0WIN+yYyQfLgqIGflgfYU8IqaZmeDFACC0jnJIwR/61UuVl11bzYRa6jAcEIYWgasIgwjjmGiFwrR80HwzZLix4LOLgPuIBa6riKftD0LcVRsz++R4RNoyYYqz5Ey5NwjOBa+M/E0KFfdKM83M1BiaRuIbcRyGEEAJaFWs+R44cSSYnJ6MKhixcuHDghUW+s3RmPSiq2YV+pIexkVtEn/JmgTG+vj65bzZEEG8VgqqJECv2Yfk2iCTCwHzDDi8Xx6rivcOgLptXZMZz3hIBZSD8EkMZIzkkAqnohiHWrxwjUQ8rod8UbYcR+zDWLN+Jyqp815r2irN/xrk/npnk4fUsxvW5WgT3BhHPvRr0Ug1CCCG6RV/Emi1uTStaMw1RZ9sSPjmIIiPvHD3mfnypkNflRVrLNmaBrUIfJdxDYGyZ9wdxhXhKl9AvAwIMEUMoGYbj6p1vZuaKMatcVTixf0LWyoKRhIGUZ2DGgtH6jenr9X/veSnTu0BVy7bWphPNwb1ssrBDW2HERSAM8WLj5eW7HFNNtQx8X31PoXnXQl74cX6u5sFzlrHBvdFEjhBCiDStizXLW7OGdy0PxBrVI2O3bwNb/+f0S1YHf5yHuVF9jb6F1gfC0MKrhqFFwwMWqvxWBAYHwgmvFQYiAjDG6A2trRYD5xhauDsW+kqOSJW++tDnb0zuTy7L8cZwjHThFtE9mGDAC9YE3PMmw4irwvERjYx1it0wOZIXshgDYpDrxCSMwWQPodRpxvW5mgfe26rPWSGEEONB62KNkEZEFwIMIRaL740r87kmmL9svfvhpWx56Id5mJuF7BCOlIYZcsIUmf2vsoA0ifEYasys40HCAIn1JDC7jCejLBiLhD/iPagDxRIQllU9KRRd4PNcA84naz8IWbZrqrCJaB4EDGO4iXyvJsOIm4RJEb7vNplSZykLrheeIcs1RRSy37Q4HdfnagieA3jSeM52bWwIIYToFq2LNRNceNjKQhgkn2W9tn5BCXNbc+rcm3YFf5iHuRF+RMls+vfWu8cFBYbpjIj4yM2UY0jEQG6MhXjREHi8VhZEoj87Hwu5OE15QDBgMdLTVe6KsHLmJhiZLcdLmAXGcdHyAmJwhJatqIJNANQJI24bJg0e2vPH2e8vf1eZSMArz2SL9RXPmoVTw7g+V0OQk8bzAbHc77BYIYQQw0erYo0qkIitE088sfdKOWxRbbxz/WLL7oPuB5dFgUM/yqPQCEOij+ue2NfrdeIKY2CkkoPC33lFEJgVxmuG94wQnrp5MByL/WTlsmXBefC5pvKKAO9AuspdHmxHWJlfhZL+EJaZlZvG+dYJ2xTtgVCpItjTMC6GLbyNcclEAgITIVH2e4V3jj5z7ew62nd6XJ+raajyyHWps8aeEEKI8aJVsVZ3nbVBrNN256Zd7gd3zqKHgz/Io9A+f+VG18cV63e6PmOUYWTd9eQRN+ObNdvLdrc+NmPMIejI0UIw1QUDhqpyZcEL0Eb+Fx6+dJW7EFwnRFeoOAlCLU+QIfAU/tQ90stWVKFqGHFXIKyRfDb6wDjFcx47ecEzgc+wPd8Lu5bj+Fz1YQKHddO4NsOw1qIQQojuILGWgkWA+cGdWDwV/EEehfaFqze7Pl5191bXZ8IIlz7yatCjxMw4RhdGBuKD8L2m862qrK2GQYlobGsdKTwL6XXSfDC+rtpwONew53pt/124OhzX1F8wWwwexr6/bEUVGBdlwoi7Dh5zm6Dh+0CoL33MA3HHkgd/+svMUhVcz3F8rhqMK8YEz4u6BV2EEEKMH30JgzzhhBN6r5TDioz0Mwxy4cop94M7d8nW4A/yKLS5S7a5Pi64ZaPzav3HtEH1H3fOhC8BngHEE94lPG6IirZEEcIPQ7Bs7oaFbLYFXjX6z2x4CEQaxlee4Ur+Ttbab+StcG2LDF/RP7inoSqGZYgJIx5G+H4Suofnndw0vn95uakIPLzebMe/4/ZcNXgGcL0QsEIIIUQVWi8wglBDcNUpMLJ2bfXFRsvy7evXuR/cs5c+FfxBHoV29tIdro8s4nrRtJig3PwLr33gZs0xNi1nBSHXttFZZW01Zqo5xzoekBg4jl/lzsAAxWsWIzCZUffz2XwQg1xjMXgYS7ZsRVUIn80LIx4VCOPDc4jXDGEaek7wfyYz7tz+hpuU+NYNMyGC4/JcBQSaX3RFCCGEqELrYo1FsBFclO4vs8D1oEr3n/Oje9wP7rk3PRv8QR6FRjU2+vjlH08lX1nxQnLttMGFUYF4YPa8n6E6GLdljRm8H8zc9wO8jRibdo4IL8vJiYE8P7xrofw3RGBeqKXoH0xOVKlGavC9KVOYZhRgTPN9IOSR7wheaDzGBqKVvDdCrOf95Mmxea6es2iVE6pM1IzTeBBCCNEOrYs1hJZ51xBsRR42BB2eNBNq/QyBBH5saaEf41Fq9PG8W55L/mVarC2fFj5te6lCVFlbDWMQr1o/jSBbQ23zvj9VqhRI1cxQqX6MWfarPJbBYstWVL0PeJYQK3UrSA4zfJdtjUWECoVG8K4RKvmf09/xeTf/dqQrQVrjufpPi3/lhOuohcIKIYQYDK2LNSB3zQQbjVL+iDBEmbXJycnZsEdrZb1xTTBOnrVz/ucXs2ssMQNOSGI/hUOVtdWayCuqwi+ffTs5/7YXkmcyyvHngRjAmA951wgBVRnvwUIoX9XKjRTg4N7WWVR6lGCMU1QHrxLCje/40wffS/75ludcmOBYPFenf0OEEEKIpuiLWAMEW1qM5TXEW7+FGoxbzpqBsYkQwsNQZY2lsjDrjJFbxqPHthiA/fZEWTGQpY+8UjkniSILIe8a4ZWEkYnBYMtWhIR0EYxHPMN4kcSnwdOIWOM7+y+37Er+efm0oLnxmeAzaRRa6LkqhBBC1KVvYs1AtOFJQ7ilG942FsLuZ45amnGoWvbF6x53fZy/bH2v18dBCNkaS3jc8Lw1XaofCB0jR64MVl2un3A97DoAgpbzKBvihHctJDTZDwb/OIfQDRI8QCERXQTfCcZFlc+OGx9+9I/k2yuedmHXtNAzaRRa3nNVCCGEqErfxVrXuXb1zA/uKK8HNJGxHlAaQrys6putsdQUZddWs1DCfnrVEFIISnJxDF7DwK+yRhpCLyQ2yW+pU9xCVIMJg6ylFfLAs8q4GEQ47jBB1UiuEc+Pb93+22TO5Q8mE//7SPCZNAot9rkqhBBClEFiLcWdm2byDuYsejj4gzwK7fNXzpTRXrF+Z6/X+WCc2hpLVEKkJHWdAh94JTDgyoQTZhXpaBMEJS1tzJu3rWyuWVYYJ161soVWRH1YfiFrWYUsGAs2LsSnoaAIXmg88zwrmJxg3Ou5KoQQQlRDYi3Fo8/+3v3gjnLlsjMWrnF9XLvt+V6v42G23Kq+EQ4YWmOpCMIs8TLFQl4RYqZKXlFV8JzhPckSlJavRP/LQL9DXjny1rQeU/+gCAaCouzYxVPEuOjnWOw6VlSEKpDkvLKsRjrnVc9VIYQQohp9F2usn8baa1SEtGIi5Kv5UN6f3LZB8N4Hf01OuWil+9Gdd/Pu4I/yMLfzbtmTnHThTP/eere6OMBAszWWEC3MoDOrHgMeujIFTDCOy3pA6oCYxHNW5D20SoCEaMbCNcKgTe+7yuLgohoINMIfd5as7GmVU/sZittlKEqEMGM84/l+/MB7mSJWz1UhhBCiGn0TawgwX6D5LS3WbDuE3SAgQZwfXXIQQj/Mw9zmLnnM9a3JimWssYTHAeHir7EUgm0Jj4oFg7pKXlFVOHfEJ56zGNiefpcJC+VapfOdCA3F6K1SaVKUA+HPhEEZ7D77iz6PI/SfiRm+kwhXQqJjJ2n0XBVCCCHK0xexhuhKizMqQuJhS4s1yvVTFdK2LVpEuw0IY+GHl7CW0A/zMLezLnvA9Y0ckqaxcCgKcGDYUjgjbdwSAlimoiN5Rf0qjY6ngPPGY1YGDFbOM1ZoIewQZulrQx5UPz2I4whjlHtcxrPLuCDslxDgcYRxzbi0tdMIg65yLfRcFUIIIcrTulijDL8JL0SYX5YfIZYWa4YJObxs/YYwFn54CWshvCX04zycbW9y8vzbXd8Ov9muKLA1ljCMCZFCxGH0lfFOYCAigvqBFfkoGxpnUDWTxZVjPYBUf0zn7ZH/hkEs2oMiNWWuMeMCgdJkJdRhgRxKQnO/PrnP5afy3cgKc4xBz1UhhBCiPK2LtZD3zMgTa3jYTjjhBPf+7t27e6/2j+8sfdD9+I5SyI6F6px/xX29XrYP4gXPGOGR35g2+r4zLYhi1hSzvKJ+GMl4ujhW2eqOPpwvfUS0xUDeEyLAD7dkH7w27qF2bWHXPNYrxLggZLfMEhPDDt9NPOJMXBAqirhtcp1FPVeFEEKIcrQu1hBbtFDBkDyxBpOTk+59Qib7zaadB9wP8CnfuyM5b/koLOS6N/nsgrtcn9Y9sa/Xy/5y9bSY+fHGw84QxLuB5yxrph7h1A8vE94+iqRgoNaFfVFh0BbQLoJw0LR3jVy2MmGiIh6uKx6iGLiXeHXH4V4gxihww/cAzzcecXJL20DPVSGEEKIcrYo1BBpiKyuUsUis4VHLe79tLrj2l+5HmPVzwj/Uw9MmrnrE9eWri+9PPv5H/4tY+Gur4UEi5I+wQV4jB8Y3DqvkFVWB88B4/+nm5tbMothCbDileXr8apL8jZdPNIuNvxgRYl7SMstLDBv0kTFKniT5k3wH8GLzetvouSqEEELE06pYKxJjdd9vm10HXnc/xCdddHty7rLhLTc97+Y9szkV258/3Otdf8laWw1xg/cCgcPMPh61tbviPSB1IGQR713TBiqCC7EZE25HmFm6r3h0KGohmgOPZaz4YjvyLPshXPoNY5NrgXBlSQxCPPu9FIGeq0IIIUQ8EmsFLFw55X6M51z+YPAHexjanCtmSmZ/98aHe73qPzFrqzGzj3D551tfSG545OVWq+9RwZHS43j62gDPIQK0KP/MvIi+dw0Bx/pVohnIPaNIRsxyDEwclKnsOQzQf0Jz6Rc5eBS3iV2aoi30XBVCCCHiaFWsWSVICoWEKBJjU1NT7n2KlAyKV9/+8+xirnOXbA3+aHe5ffG6J6bPfYU7/wOvxuVSNU2ZtdUwlq+fetmJKT6DyMMr1+TsP5UpEUix60NVxRbXLjL82c7Pz7PS/nUq79Vlz8EjrtT68od+k1x191ZnkBLqdeYP7nCNYgq8xntsc//Wve4zXQwFw1NGqG0R5FAisLn+ww5jh3FOqDFjiWvQdlhxGfRcFUIIIeJovcCIVXREmKUpEmu8zvuItkGyavMe96N88vzJ5OylO4I/3l1sX75xpztnzn1y4zO93vSf2LXVQh4QDEzyaczgLLsGWhr2x77aKqCQhpCzopA63iNPzTemMbIfP/Be73/t8+FHHydP73sluWbVtuScH93jxkyVNnHJXW4f2/YccvscNIwlxlSRAMOrS2hgTKXSLkP4LOG9jHHGHWNokKI/Dz1XhRBCiGJaF2tWun9iYqL3ynHyxJoVF6H5a7MNCgxQfpypYjYMeRbzbt49W6Vs0c8293rRfxAieLFiytEjbLLC/ygQQSgXnioqLlYpKY4hjhHL+lH9gv4T2lkU1miLDhssd4Cx3TZ4Ba68a4vzljFWrH12wc9dmNfE4qnkC9dsSb50/ZPTRuozbn0sGn/zGu+xDdue+sNPijz2ednkowP1PCD0iyp9EoLKuBjWPEE8xGuffdsJfr4feKXb9ho3hZ6rQgghRD6tizWElnnXEGUxi2Ij1OwzlO/vAoR3EfbFj/RpF9/T8UVd9yanX3KfO1cqrx372+AMt9iFnvGAYDDHhKBhVONlY3uM8ZjwLvaLaOynt8ogDJJwTozoLMy7ZuvKWS5bW0Y3YWiEMFooGiFdp128ygkvhFh4XBW3c2961lXIO/2S1W6fM/u+zeUo9XvBYBNheSG0TCJwnQkZHCYYUybwraJqXa/zINBzVQghhMindbEG27dvn/WS0RYuXOjWTqPxf7xuCDT+z9+2HX+zOHZXeP/Yhy5vZ8awWNXJmWBmfmcM5dtcONs7RwdrwFEaHKOyCMQXnrUyYIST74WnzTwKIbGHYcs2eB8GBedFPlSeKMCbRhEIAwM8T+BVgfFw7erHZ0XaSReuSD53xUY3bkLjqU5jn5Rnt4p5HBOBiFDsB4SS5oXfMn4YN7Hr4nUBvMI2UYHHljHT1TDHWPRcFUIIIbLpi1gDvGjmLYtpCLouCTUD7wDFFfjRPnn+HcnZS58K/rgPop3zkx3JKd+/c9ag6LcnI42/tloehCeyXZ0iIngVLFcHgWgeKjxWhBN2obqi9TMv3A6xZmu00SfERFNs2X0wOWPBTK4NXi8q8fXDMOYYcxbhPZnxtJ06f2WyYUdcwZmq4G3FY5YlZBgXlK6PKTwyaPA6U8ERsc/4qBIC3HX0XBVCCCHC9E2sAeKLYiFWOCTdWDybHDcW0+4y733w12T+spmyzRigE1dvDv7I97ORO2RJ79++fl0nZn6z1lZLg4cgpgBJDAhD1moj7JBqkj9c8/tpsXbYGeddAM8IRndW6XSEGl5AO1/EWhPhbRRCmBmvtyVnLFxTK9SxauOYZ146syAy7adrn3RhcG1AeCCiJgsEPa0r4yINExeMY9YeRHRSpKdfRXEGhZ6rQgghxKfpq1gbJTAyb1izfdbwxAg996ZdwR/8Nhtei7Muf3D2PAgz60r59Ji11WwB6TpetSxu3fZa8h93vph862f7XUgc+XNdMM5ZiBgRltVnPD4WOkoYZB3vD3k1tqaVM4AXTwXHUT8bRrh52TDOCYNrEu4z+X9ZXjWED9e4yOPbbxibnDsi0vIx8RB3VVC2gZ6rQgghxCeRWKvJQ0/un83/OXn6X6rizbu5/SR5jkE+0EkXHc8HYq2rrhC7tlqRB6QqCCKOT64YRjviBy8FXi28eDHVKduECoUIhpAhjsBFbPAeBUbywvnyeOvdD5KvXb1mZmzOn0y+dP324FgaRCPMjXA3zo3wtyZDywgVzMqTZKwhlGMK2fQLJizI1+Q+MyYYu21MXgwTeq4KIYQQM9QSa4QrDkPYYttQMIEyzvy4ux/4acP4nxZvaqWy2XnLn3PV9szQpeE56drCrDFrqyFKEE9NF0jAO4HhG1ozCxGJYUzu2FUbDrsCDYPyXOBBuenXr/b+90kQsSY4yLnjPMuAR82EGiX1qdIYGk+DbHgvKCjBOSLYCIOrCwVcEGOhe8o1ZLx1YS01cs4QjghLJhXIScsKjR1X9FwVQgghaoo1K71v+Wbko3VhTbRBwQ/78ZCzmUp75AcR9lWn2h6fJXfizEt/MTvjS6Pk9f6XB1fhMAsMZcRSkfcKT1dMpcgyHHz7WGERD0AgYtgjijhXPF39NuI5BzwpIVHL+ZtHjeUGCOOMhXCti2/d5MbIqT+8u9Pl0Dk3SrbbeK4Tasa4wyNpBVp87HoOsrw99xLByL0kzJF8zphlJ8YdPVeFEEKMM7U9a36BEGtUcqQU/7iy68DryYJbNs6G8VjDi0CZdIyMuUu2uVAwPB7M6tL4m9d4DyOCbWfW9Tm+XhX7ZN8co6vErK2GQW2hfk2BZwLPSVkvFCINTyDGPOeNiGva25cF4Xhch9D6bxRewfvCuWDcx1YAXLbuaTdW8BJ00aOWbnjYzKPBIslVQfj7Sx8Y3F/GRUjE9QOrVMokAl5S7nW/xtcoMe7PVSGEEONJ7Zw1q/Dor49mjVL9LGo9rmGShKJt2nnAhfKc+YPj4TVlG59lZpl9NREq1jaE9+V5zBBoGNVlRVUeCBnC3+rkv3FenBPhkRjW/arAh5gIeQOt+AqGPUsPxPSNXJ+ZcbPCGaghcdTFdvbSHdOCbabq3t2PlJ/o4RpxrdKeKhPDVFbsJ+Qasq4fY5JGoZi2FjgfN8b1uSqEEGI8abTACCGQiDNCItPCDTGHqOvi2mn94MOPPk52vviaK6FOtbPLJh914TbzFt3rjAYaf/Ma77FwMdtuf/6w++ywELO2WpYHpCoY6oQTll1UOw9COAmNxCNj4ZptekMQaly3dCgmwhejn/eLrtkb77yfnNpbgBoPQkgUdbnNXbLVnTtejrIFRxCyaW8uY5B7x33sBxwPrxneM+4l3rRBhl2OA+PyXBVCCDG+NCrWfAiDpPhIaCHscQ+THGWK1lazvCJCJZuA/SFoaE2GVBrsk3MlzwgDnDL6eLzaAEOfa+NXKiS0k+NSHZBCFHnHvvKuLU7snHXZA0ExNAxtzqKHXB/IuYuFa8M18oUR9417Rvn7tsGbx5gnVJVj4p1VmKMQQgghmqA1sWbgSdu+fbsTaGnRNu5hkqNI0dpqiLmifLYy4E3Dq9YP45gwNoqB4G2jn4TWNb1WF/tP9wchwOu0LO8hBREQOSdduHIo8tSyGkUfTu55B2Pzh7gu5Pf54NVinLUh4AERTQVHxgIeTzx7sTmFQgghhBCxtC7WfAiTzMpv4zVE3biGSY4CRWurIUBCeUVVwUAmH6jfRjICgMWK8ebhTWk63I1y/r5HyLxrv3/rmLt+IQFCmBcC53NXbAiKoGFqlGenL9+8Zm2vd9lY2K3vcSQ/jHHRtJDGg8caaIhp7gM5jW15WYUQQgghoK9izQdvWlZ+G+GTCpMcPvD64OXIAiO6TAn6PAg1w6sx6LWpCFmkX4jUBff/znkO6y5ojBjDK4TnxiD8kuvL6+kQ0m17DjlxQ0XFfiwc3Hajgt8p37/T9WnDjvyF1bkmftgtoaQIqaYWvTZhjnhGmPMv178tj50QQgghhM/AxJpPVn4bQg5BN85rtw0LGK94OLLWVkPAYOw24YmwYhysqdYlnnv1g1mjHgGRru5YBq4X3iE8OYD4YL/r9/zRefR8vrP0QSdsWNQ3JH6GsVEgpci7xjX5+uS+WcGOx5Zr1EQFT/ZhyzngSSPkta4IF0IIIYQoSyfEmuHnt/nC7YILLuhtIboKa1jl5aLhcWui2APVEjGgmypQ0gaE5j00LaqsbDt/VwnVRITgPUQEAl6kFdted4LE9vfO0WNO1LBQcJcXvy7b8K7ZQsWvvh1e4gFBjMcREFdcF7tWVeCaElpLDhrXHc/moD23QgghhBhvOiXWfPC2WYikxFr3ocBD1tpqaQ9IVdgP4YbmbRoG/EqBiNWy+Xp45xCniFTLz+JaE24Jtq7a6ZesDoqeYW5UtaRvoXXXGEuMKcYEjWsUWli8CPIoCanlmnKPuFdN5VQKIYQQQtSlU2ItKxwST5voLogIDN2sgg7pvKIqsG8qMPp5XMMEIXQILPLaEJzkuSEyYkBM8Bm2x0P5v+sPufXDYP6y9U7QTFy9OSh4hrlZKCRhnmkQvoQpMi64pqxFVwZCaPHKIX5ZFw2h13RBEiGEEEKIugxcrFmhkdB6bCo0MhwQOpYlxvCAIORihUkI8uEoTEKVxFGAypG3PjbjbSP/jAIWRQUrEHcItD/95e9OYHz75y8mB9/8wC0gjaA596ZdQcEzzI2wTpYioH9vvXvc20XeI9fu6F8/cqG3VOOMgaUXrFIkDYHHa0IIIYQQXWUgYk0l/EcLcnyyQscQcXjW6oCwwfsxahX48ORQuAKPITlSeM3yxAPXgXA9tpt/34Hk6gdfdELmtItXBcXOKDTCO+njuif29a5C4oQ71wDxzt9544IwR7xmjB9CJZteZkEIIYQQok36JtbyFsdW1cfhBS9H1tpqFH3AC1SluIaB94Mwt1EPUSMsDyHB9UKAhMrD8/+rNhxO7tz+RvKtn+1P/m3F3mkhsyKZs+jhoNAZhfb5Kzc6sbZi/U53DZgUQHTd9eQRJ3KzxgXbmfeS60koaT8WThdCCCGEaJLWxVpWHhpNYY7DD14zvBwhzAtUlabXzBoGEBQUaiHkkb7fu+PNTyyHYDlaSx95NfnqbXud52li8VRQ6IxC+8LVm51Yu+rura7/hD3S9wtXHfjUuCDklvHG5AHeXsJz60wUCCGEEEIMmlbEGh6yrAWvCXMkBFJhjsMPnh48QaG11fAU8V7VtakowY5XhCqI4wqeSSuCgUcN7xDXnFDJ/7z7peQr02Lti9c9kcxdsjUodEahzV2yzYm1BbdsdN7G/5i+Fv9x50x1TEC8Uh2UtdAQtxQdaWItPyGEEEKILtCYWEN8ZeWh4VVDvFFMRIwOeWur8TqejSrYmlkqoT4D3rbtv/uzu6YINwTJ0wffS/55+XPJecufT85e+nRQ6IxCO3vpDifWWBz7olUHkm9MTo+L1z5wRVnIWWOcUBkyFDYqhBBCCDHs1BZrWXloNF5XmOPokrW2muUVVckRwmvEZxEn4tPgUUKsIdrOu+mZGVGzfHQWw043qlwi1r7846nkKyteSK791cuuGAueNIqzVPXcCiGEEEIMA7XE2v79+z8l0Ah9VJjj6JO3thqFH7IWyM6DfVFSnfLqIp8PP/pHMvG/U8l5y/cGRc4oNcTavJt/m3zt9n1unb26i6sLIYQQQgwLjYg1whwpFqIwx/Eha201QiMp/lA2JI3ty6yZNa6QC0hRFzxr/3zTb5I5lz+YnHvTs0GRMwrNPGvnXH5f7woIIYQQQowPtcQahUQIgxTjR2htNQQXlQqrhDCSd0RYpfKOPg2hoQ/t+aO7tlQ6pOIh3qVvX7/OCZmzlz4VFDqj0PycNSGEEEKIcaOxAiNifMhaW43QR0RcWShPT6n6rDWzxhErKkIVSMJNWTMsLY4XrpxyQmaUq0F+8brHXR/nL1vf67UQQgghxPggsSZKE1pbDY8Y4Y9U5SsDRSJCa2aNK/vf+MvsYs6L1x9ya81lFWq5dvWMkBnlddYmUuusCSGEEEKMExJrohSIMvKl0murkcOWVcY/C/LbqOw3zmupAdcS8YtotQIrhD4WceemmXyuOYseDgqdUWifv3Kj6+OK9Tt7vRZCCCGEGB8k1kQpQmur4fmh3H6ZddEOvHnMfQZP0jhCyCdho7Z2GotfUzykDI8++3snZE67eFVQ6IxCO2PhGtfHtdue7/VaCCGEEGJ8kFgTpQitrbb22beTa371cu9/xeBJw6P29MHxW0vtuVc/cIs5f31yn7uWiN8q69HBex/8NTnlopVOzMy7eXdQ7AxzO++WPclJF8707613tUC6EEIIIcYPiTURTWhtNRYl5jWKjsRAbhqhfpuef6f3yuiDOKWICgKVNegIGeVaNgGFNxAz5HaFBM8wt7lLHnN9UyVIIYQQQowrEmsimtDaaixSTNn9GBB5lz/4h+Tup470XhldEGMIUqpcEu5Jn1/5U/O5eYQHImgIFwwJnmFuZ132gOsbuXlCCCGEEOOIxJqIJr22Gl4ywvlY86sICpPc8Mgrro0q9JGwRvqItxER+9tX3m917TjCAxE0hAsSNhgSPcPZ9iYnz7/d9e3wm58MuxVCCCGEGBck1kQUobXVKOGf9rRlQQENvGptCpdBwbXhWlAohD7+ev+7Ljy0X3xn6YNO1IxSKKSFQJ5/xX29XgohhBBCjB8SayKK9NpqeNPwqsWsj/bQnj+6PLV+Cpi2od/0C28jIpZw0BgPYxts2nnACZtTvndHct7y54LiZ7ja3uSzC+5yfVr3xL5eL4UQQgghxg+JtT6y5+ARl2O0/KHfuEV+v3vjw8lXF9+fnPmDO1zDi8BrvMc292/d6z7z8T+OF/QYBKG11fCoIeCKoOIjOVvpddmGEao2bv/dn13lS8IcuQZllitokwuu/aUTN6xLFhZAw9MmrnrE9YXvxqDHvhBCCCHEIJFYa5EPP/o4eXrfK8k1q7Yl5/zoHmeAVmkTl9zl9rFtzyG3z36TXluNQhmIt6KKhqyhVqZSZFehHysff931ZfH6Q8njB96rXG6/LXYdeN2NlZMuuj05d9nwlvGfd/Oe2Vy17c8f7vVOCCGEEGI8kVhrgQOv/jG58q4tzlvmi67PLvh5MueK9cnE4qnkC9dsSb50/ZPJl298xhWGoPE3r/Ee27DtqT/8pMhjn5dNPuqO0S/Sa6vxfz8kMgTl6hF0FNgYRt754O9u/bgLVx1wIZzrdr/tXusyC1dOuTEy5/IHg0JoGBpjnj7gYRZCCCGEGHck1hrk1bf/7EIYbaHi//PdFclpF69ywgshFjJOY9q5Nz3rQsNOv2S126cJN4zztivl2dpq5kk6+PYxJ8Ly8s/I52JNsfTi2V2HpQU4Z7yI9JGiKAfePNZ7t/sw/mzszV2yNTiWuty+eN0Ts+O7n5MRQgghhBBdRWKtAd45eiy5dvXjs4bySReuSD53xcZk3s3Nh6OxT/KSLFSMYyIQMdTbgLXVbn3seMVHhAyvZYHgoehGkeetSzz36gcu/wxRiteQPLuuhTnGsmrzHjcuTp4/mZy9dEdwDHWxffnGne6cOffJjc/0eiOEEEIIMd5IrNVky+6DyRkLZoxMvAKEoPUjZ4hjzFn0sDsmxz51/spkw45PltZvAoQXOVtAMQ2KhWQJGQqRXLXhcHQ5/0FC5UYqOOIBpI8I0KIcvGGB/EYn5L93x1DkrzEBYdUfF/1sc68XQgghhBBCYq0GeABmRNptyRkL19QKdazaOOaZl85UAqT9dO2TjVXQozAIOVsGoiYvtBGRhljr6lpqhG4+8sKfkksfOOhE591PHXHFUkYN7j85X4yH0y6+p+OLZe9NTr/kPneuVLQ89rfhrxoqhBBCCNEUEmsVwKC0Yg54tshJCxui/WssiGxetvnL1ifvH6u/5hel+SmyAVSERLhlCTHCHhFzhEF2Cc732cNHkxseecWFOf508yuu6ElXBWVTcP8pfT8j2FZ10sOGR20mD/M2Vy2VcGIhhBBCCHEcibWSvPXuB8nXrl7jDExybL50/fagITqIdvbSp6bPaaYCJWu21Sk+gpihyAYVEPmbioisMRYCbxvhhDELZPcLvIKITTxolz/4h+TX+98dqUW5Y+D+Mw5mxuodbnyExs0g2jk/2ZGc8v07Z4Va24VyhBBCCCGGEYm1EuBRM6FGSX2qNIYM0UE2PCh4UkywvfdBtTXO8KSxphggxvCahcBLhaijVP+gIeeM3DPO9b/ufcnlpJGbNs5w//G0Mh6cF/jqzcFx08/G0hRWTOTb16+TR00IIYQQIgOJtUjIA7r41k09oXZ3p/OAODdylThXcpeq5LBRFdEWfyb8EfGWBu8VoYVWgGQQcH5Ub7zmVy+7cyFvjkIo4jjc/xvWbO8JtttcjuO5N+0Kjp02GxMJZ13+4Ox5UMW0qfxKIYQQQohRRGItkmXrnnYGJuFkXfSopRuGsYVEUh2wDP7aaniqKNef5q2jf3chhlmhkW3D+mcrH3/defXwAJqwFNk89OT+2eUlTp7+lwWo593c/qQDx2C5iZMuOr7cxP1b9/bOSgghhBBCZCGxFgFGLkYmYWRdyvspaqyzZeFmdz+yu9ebYmxtNcQPgiztqSL3ixy2h/b0d+Fi8ucoeMKxaet2v+1eE/GwHh/l8WfG87Rwmh4f/7R4Uyue4vOWP+cWc7dJAxqFebTgtRBCCCFEHBJrBbzxzvvJqb0FqMm1CRmlXW5zl2ydMcovWhldxIGcL0IbqfBIKX4fio1QsGPyiTd6r7QL1SXxmuE9w4uGNw2vmqgHgul4RdOZhdxZfoKctjqLufNZvidnXvqLWU8ajXDc/S/PVBYVQgghhBBxSKwVcOVdW5yxedZlDwSN02FocxY95PpAzl0RtrYa3jNCIfm/DyXwaW2XvsebR/4Z50A+GnlpCnNsnl0HXk8W3LJxNjzSGkVqPnfFRife5i7Z5jzKhP/iLaPxN6/xHuKMbWfWS5tZPoLGPtk3xxBCCCGEEOWRWMsBTwBG50kXrhyKPLWshrfj5J53sMhwtrXV7t3xphNlPiwijVetrbXUqNxIBUeWAcC7Rzgm+XOifah0umnnARcieeYPjoctlm18Fo8d+6paiVQIIYQQQswgsZYDoVsYoJ+7YkNQBA1TIy+JvnzzmrW93n0aW1sNb9rXJ/d9ouz9puffcXliTa+lhgePNdAQgeTHIQjT3jzRXz786ONk54uvJZMbn3FVJC+bfNR9F+YtuteJMRp/8xrvXbv6cbft9ucPu88KIYQQQohmkFjLYNueQ07cUByhHxXz2m6ErtkixBt2vNTr5SextdXIRyME0SAEEW9XU2upIQpZn+2nm19xYY78++zho62HVgohhBBCCDFMSKxl8J2lM+tBUc0uJH6GsZFblOddY221h/f80XnVrMoihUbweDVR1OOVP/3Nec7YH560R174k/OsCSGEEEIIIT6NxFqAd44ec6KGCnldXvy6bMO7ZhX6KOHuY2urLdvymstbAzxpeNRCC2LHwn7JPSMHjX2Rk+aHVwohhBBCCCHCSKwFsHXVTr9kdVD0DHOjqiV9S6+7hqC6btPLLmeNvDQaVSHJVSsLVRsJncRThwAkpDK9VpsQQgghhBAiH4m1APOXrXeChrLlIcEzzM1CIQnz9MHz9aN1f3Brq1Ht8dIHDrqKkGU4+PYxl++G4CP3jfXR2qocKYQQQgghxKgjsZaCEua25tS5N+0KCp5hboR1shQB/Xvr3RlvF9UXCVHEC3b0rx85jxhFP2Igt23d7rddpUgaZf8t300IIYQQQghRHYm1FFt2H3RChkWBQ2JnFBrhnfRx3RP7XJ/JUfuvew+4UMiVj7+e/M9Dh3IrMxLmiNcM7xnFQvhMEwVIhBBCCCGEEMeRWEtx56ZdTsjMWfRwUOiMQvv8lRtdH1es3+lE2Tfv2OdCF3+5a8ZDlhW6SN7ZrY+95jxw1/zqZZeXhnATQgghhBBCNI/EWgoWAUbITCyeCgqdUWhfuHqz6+NVd291lR6/MS3WVmx7w3nJ0iGMVG4kj+2/7n3J5bXhfaPCoxBCCCGEEKJdJNZSLFw55YTM3CVbg0JnFNrcJdtcHxfcsjH5718eTP79Z/uct4x10ADP2q/3v+vWQkPAsTYaeW1CCCGEEEKI/iGxluLb169zQubspU8Fhc4otLOX7nB9/PqSdclXbnsh+bfb9yW/ffn95LevvJ/c9OtXnXCjwMizh4/m5q4JIYQQQggh2kNiLcU5P7rHCZlzb3o2KHRGoVHlkj6e95Mnk/OnxdqPN77sqkHiSXvkhT8lf/lQYY5CCCGEEEIMGom1FIgYWkjkjFKb6ePe5F9X7ktW73zT5aYJIYQQQgghuoPEWopx8qyds+j+Xq+FEEIIIYQQXUNiLcU45ax985q1vV7//9u7/9cqqzgO4P/btvBrthTNDaFfgyBIV4Y4NGWK4Zcy1Cltzoxs6so0ZcwvZNMVMr+gk4zRDwsTzEIr0SzlyXPus1uMR93M2Xmurxd82GXbvfd8fnzznPM5AABAaoS1MZ6FaZBzNpyIPbZsOZR3DQAApEZYG2P9nkqQqeV71hr/dc8aAACQJmFtjJ19lfNcM1ceLAw6tVCzVvXGHjsODeZdAwAAqRHWxjh65vsYZKYs6S4MOrVQ01p7Yo+f9V/MuwYAAFIjrI1x4+btrGFRZwwzTe3nCsNOmat56/msbmGlv5+u38y7BgAAUiOsFQiDN0KYCWe7igJPmWv2uq9ibyZBAgBA2oS1AmF7YAg0YbtgUeApc81Y/nnsLZzNAwAA0iWsFQjbA0OgCdsFw7bBotBTzhrK6lu2x95Grl7PuwUAAFIkrD3AaxsPxFBTS1shR7dALmjbnXcJAACkSlh7gL7B4RhsGt7YkTVvu1AYfspVQ9lziz+KPe0/+W3eJQAAkCph7SFeWb8vhptwL1lxACpPNa45Ent5efXe7O69e3mHAABAqoS1hzg7fCUGnLpF27P5W8o7xr+p/Xz1rNrAxZG8OwAAIGXC2iO0dh6OIWfmigOFQagMNbOtchXB65sO5l0BAACpE9Ye4fK1X6uXZM9ed7wwDKVcczacvL/2jrj+4cs/510BAACpE9bGofvY+Rh26lu6srkbTxWGohTrpU2Dcc1h7V29p/NuAACAMhDWxmltd38MPWE6ZBnOrzW1n6tOf1z54bG8CwAAoCyEtXEKExTDma8QfqYs2ZX4ZdlD2dSlu+Naw0TLW3/8mXcBAACUhbA2Ab/fuhNH31cCW3eST9jCE7WpS/fENc57e1f2y2+38tUDAABlIqxN0MjV69mCtspTq/qWHdncjd8Uhqb/o+a9fypreHNnNaiFtQIAAOUkrD2GGzdvZy1bKuPww6TFxneOFYanp1kvrv2yOkzk1Xf3e6IGAAAlJ6w9pnCG7b2egTywfZBNX7Yvm7/5bGGQmswKWzFnrDhQXceaj4/HtQEAAOUmrP1HX3x9qXoPW/39n+EC6qb2yR8+Er5j1qrerG7R9vjdYQ17jw/lqwIAAMpOWHsCwsXZYTz+6NOthpau7IXVfZMyMbJ524Wscc2ReF5u9PtaOw+78BoAAGqMsPYEhcAUgtNoiKpb2JFNa+2JZ9rClMai8DWeCu8NZ9KmL/u0+iQtVLhK4NIP1/JvBwAAaomwNgnODl/JFm/trW6PHK0w7v/5tt4Y3mav64+TJOdvPhOfloUKr8Pvwt9COAv/W7kvraP6GeEzw2eH7wAAAGqXsDaJwmXUfYPDcYvk9Lf+2bY40QrvDU/swmeFSZQAAEDtE9aekjt/3c0Gv/sx6+o9HadILu86GrcxNq38JIaxUOF1+F342/o9J+L/Dlwcie8FAACeLcIaAABAgoQ1AACABAlrAAAACRLWAAAAEiSsAQAAJEhYAwAASJCwBgAAkCBhDQAAIEHCGgAAQHKy7G9QgB00yaU2BQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "image/png": {
       "width": 515
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(filename=\"assets/vector_to_norm.png\", width=515)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，上述网络的输入是 5 长度的向量，通过 MLP 过程，即两层神经网络层 (线性层与激活层的叠加) 与一层线性层后，得到长度为 1 的向量模长。第一层的神经元数量为 4，第二层则为 3。\n",
    "\n",
    "网络构建完毕后，我们可以构建用于训练的数据集 `X`。\n",
    "\n",
    "- `X` 的第一维度为 6，意指总共有 6 个数据点用于训练；\n",
    "\n",
    "- `X` 的第二维度为 5，意指作为特征 (Feature) 或者说数据点的向量长度为 5。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.25963, -0.17396, -0.67875,  0.93826,  0.48887],\n",
       "        [-0.67309,  0.87283,  1.05536, -0.00479, -0.51807],\n",
       "        [-0.30670, -1.58099,  1.70664, -0.44622,  2.08196],\n",
       "        [ 1.70671,  2.38037, -1.12560, -0.31700, -1.09247],\n",
       "        [-0.08519, -0.09335, -0.76071, -1.59908,  0.01849],\n",
       "        [-0.75043,  0.18541,  0.62114,  0.63818, -0.24600]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.randn(6, 5)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以很容易地构建该数据集下的真实值 `t`。真实值就是对数据集 `X` 表示数据点的维度 (第二维度) 取模长就得到："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.29526],\n",
       "        [1.61155],\n",
       "        [3.16858],\n",
       "        [3.33766],\n",
       "        [1.77540],\n",
       "        [1.20463]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = X.norm(dim=1, keepdim=True)\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在当前简单的问题中，MLP 的意义就是一个拟合映射：\n",
    "\n",
    "$$\n",
    "\\mathbf{X} \\xrightarrow{\\text{MLP}} \\mathbf{y}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们定义 `y` 是 MLP 模型下，输入数据 `X` 给出的结果。\n",
    "\n",
    "我们会希望在参数学习的过程中，`y` 渐渐地接近真实结果 `t`。但在完全没有进行训练的状况下，我们应该会预期这两者的差距会相当大 (我们暂时拿 L1 损失函数来衡量误差)："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.9639991521835327"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = model(X)\n",
    "float(nn.L1Loss()(y, t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若要让 MLP 模型的结果 `y` 接近真实结果 `t`，我们就需要对模型的参数进行优化。下面的代码使用了以下的优化参数：\n",
    "\n",
    "- 优化器使用 SGD (stochastic gradient descent) (在一般的深度学习问题中，通常大家会更推荐 Adam 优化器)；\n",
    "\n",
    "- 学习率始终为 0.02；\n",
    "\n",
    "- 使用 L1 作为损失函数；\n",
    "\n",
    "- 学习过程中使用梯度下降次数为 1001 次。\n",
    "\n",
    "但需要指出，实际的深度学习的优化过程的参数是需要一些经验性的手动调整，以及引入验证集等降低泛化误差的做法。下面的代码只是非常简单的实例而已。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), 0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, X, t, optimizer):\n",
    "    y = model(X)\n",
    "    loss = nn.L1Loss()(y, t)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss in epoch    0:   1.963999\n",
      "Loss in epoch  200:   0.397570\n",
      "Loss in epoch  400:   0.049674\n",
      "Loss in epoch  600:   0.098072\n",
      "Loss in epoch  800:   0.103638\n",
      "Loss in epoch 1000:   0.021809\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(0, 1001):\n",
    "    loss = train(model, X, t, optimizer)\n",
    "    if epoch % 200 == 0:\n",
    "        print(\"Loss in epoch {:4d}: {:10.6f}\".format(epoch, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这就完成了一次 MLP 的学习！我们可以看看现在模型作用在训练集上得到的结果 `y`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.24895],\n",
       "        [1.53482],\n",
       "        [3.02538],\n",
       "        [3.18011],\n",
       "        [1.70085],\n",
       "        [1.18437]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你应该会认为这已经比较接近真实的目标模长值 `t` 了吧。\n",
    "\n",
    "从刚才训练过程的输出中，我们应当知道训练集的误差大约在 0.02 左右。但这个模型 `model` 是否真的能用于预测任意的长度为 5 的向量的模长呢？答案是不仅否定的，而且刚才的模型甚至会比随机的预测还要糟糕！\n",
    "\n",
    "下面的代码是取了足够大样本数量 ($1,048,576 = 1024^2$) 来作测试集 `XX_test`，并取另一个取自同一分布的 1,048,576 数量的长度为 5 的向量集合 `Xr_test` 作为随机向量。我们可以经验地认为，对一个向量模长的随机的猜测，可以是与测试集相同分布的另一个随机向量的模长。\n",
    "\n",
    "如果一个理想的机器学习模型应当要比随机猜测要好许多。但下面的代码显示，随机猜测的误差为 0.78 左右，但模型 `model` 所给出的误差却是 0.91 左右。这意味着模型给出的结果甚至不如瞎蒙！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for  random guess:   0.776423\n",
      "Loss for trained model:   0.911286\n"
     ]
    }
   ],
   "source": [
    "torch.random.manual_seed(0)\n",
    "XX_test = torch.randn(1048576, 5)\n",
    "Xr_test = torch.randn(1048576, 5)\n",
    "print(\"Loss for  random guess: {:10.6f}\".format(nn.L1Loss()(XX_test.norm(dim=1, keepdim=True), Xr_test.norm(dim=1, keepdim=True))))\n",
    "print(\"Loss for trained model: {:10.6f}\".format(nn.L1Loss()(model(XX_test), XX_test.norm(dim=1, keepdim=True))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，对这种现象的解释是，模型的参数较多，而训练集 `X` 的样本数量只有 6 个，因此会发生相当强的过拟合现象。简单粗暴地说来，就是模型会强烈地喜好接近训练集出现的 6 个样本的数据点，而不能很好地处理不太像训练集的数据点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 大样本学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解决上述过拟合问题的一种解决方案是提高训练集数据量。现在我们重启一个从结构上完全等价，但参数的选取可以不同的模型 `model_large`。我们更换一个数据集：这次使用 1024 个样本作为数据集 `X_large` 来学习；并且使用以下的优化参数：\n",
    "\n",
    "- 优化器使用 Adam；\n",
    "\n",
    "- 学习率从 0.5 降低到 1e-5，降低的标准为连续 5 次 epoch 损失函数大于最低一次 epoch 损失函数值时，学习率降低到 0.9 倍；\n",
    "\n",
    "- 使用 L1 作为损失函数；\n",
    "\n",
    "- 学习过程中使用梯度下降次数为 1001 次。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.random.manual_seed(0)\n",
    "model_large = nn.Sequential(nn.Linear(5, 4), nn.ReLU(), nn.Linear(4, 3), nn.ReLU(), nn.Linear(3, 1))\n",
    "X_large = torch.randn(1024, 5)\n",
    "t_large = X_large.norm(dim=1, keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model_large.parameters(), .5)\n",
    "scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode=\"min\", factor=0.9, patience=5, min_lr=1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_with_scheduler(model, X, t, scheduler):\n",
    "    y = model(X)\n",
    "    loss = nn.L1Loss()(y, t)\n",
    "    scheduler.optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    scheduler.optimizer.step()\n",
    "    scheduler.step(loss)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss in epoch    0:   2.000235\n",
      "Loss in epoch  200:   0.344131\n",
      "Loss in epoch  400:   0.308635\n",
      "Loss in epoch  600:   0.307540\n",
      "Loss in epoch  800:   0.307498\n",
      "Loss in epoch 1000:   0.307497\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(0, 1001):\n",
    "    loss = train_with_scheduler(model_large, X_large, t_large, scheduler)\n",
    "    if epoch % 200 == 0:\n",
    "        print(\"Loss in epoch {:4d}: {:10.6f}\".format(epoch, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于上述学得的模型 `model_large`，我们发现其训练集上的误差约为 0.31，明显比小样本学习中的 0.02 要大许多；但在相同的测试集中，其误差却降低到 0.35 左右。这意味着 `model_large` 尽管仍然不能很好地预测向量的模长，但至少已经比随机猜测好许多。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for  random guess:   0.776423\n",
      "Loss for trained model:   0.352113\n"
     ]
    }
   ],
   "source": [
    "print(\"Loss for  random guess: {:10.6f}\".format(nn.L1Loss()(XX_test.norm(dim=1, keepdim=True), Xr_test.norm(dim=1, keepdim=True))))\n",
    "print(\"Loss for trained model: {:10.6f}\".format(nn.L1Loss()(model_large(XX_test), XX_test.norm(dim=1, keepdim=True))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这种模型下，训练集与测试集的误差都不算太小但比较接近。一般认为这种情况已经有效地解决过拟合的效应，但存在相当严重的欠拟合效应。欠拟合效应反映的是模型本身的缺陷。\n",
    "\n",
    "在这个例子中，这意味着 `model_large` 的参数不论如何学习，恐怕都无法作出有效的预测。一般来说会有效的解决方案可以是增大模型的大小 (反映在增加 MLP 神经元数量与层的数量)，或干脆更换一套模型框架。\n",
    "\n",
    "如何解决当前例子的欠拟合问题，不在这个文档的讨论范畴之内。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch 普通前馈网络的定义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 段落目标：重复模型过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "刚才我们花了不少精力，除了展示一个比较简单易行、但比较完整的深度学习流程之外，更重要的目的是说明了这篇文档的模型与数据。\n",
    "\n",
    "后面的文档中，我们将会拿上面提到过的小样本模型 `model` 与数据集 `X`，来探讨 PyTorch 中 MLP 层具体是怎样计算，以及如何进行自动求导以更新参数表。这一段中我们先会解决前一个问题。我们的目标是，对于任何学习过最基础矩阵运算的人，都可以很轻易地理解到在给定参数的情况下，MLP 的前向传播过程不同于很多机器学习模型，在本质上仅仅是由非常简单的矩阵运算构成的。\n",
    "\n",
    "在解决问题前，我们先将模型换为没有经过训练的状况。这么做的目的单纯地是为了让梯度变得大一些，以便后面比较有效的定性与定量分析。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.random.manual_seed(0)\n",
    "model = nn.Sequential(nn.Linear(5, 4), nn.ReLU(), nn.Linear(4, 3), nn.ReLU(), nn.Linear(3, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们代入的训练集是 (6, 5) 的矩阵 `X` $\\mathbf{X}$；其第一维度为数据点数量，第二维度为特征向量长度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.25963, -0.17396, -0.67875,  0.93826,  0.48887],\n",
       "        [-0.67309,  0.87283,  1.05536, -0.00479, -0.51807],\n",
       "        [-0.30670, -1.58099,  1.70664, -0.44622,  2.08196],\n",
       "        [ 1.70671,  2.38037, -1.12560, -0.31700, -1.09247],\n",
       "        [-0.08519, -0.09335, -0.76071, -1.59908,  0.01849],\n",
       "        [-0.75043,  0.18541,  0.62114,  0.63818, -0.24600]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.randn(6, 5)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在模型的作用下，得到的结果是 (6, 1) 的矩阵向量 `y` $\\mathbf{y}$，其第一维度为数据点数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.05652],\n",
       "        [0.11903],\n",
       "        [0.10917],\n",
       "        [0.10579],\n",
       "        [0.10462],\n",
       "        [0.11395]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = model(X)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的目标是用普通的、容易理解的矩阵过程重新描述 `layer(X)`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型 `model` 的定义是通过参数给出的。这些参数在 PyTorch 中以字典的形式储存在 `state_dict` 方法下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "collections.OrderedDict"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param = model.state_dict()\n",
    "type(param)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这意味着这些参数是可以被读取的。由于这个模型并不庞大，我们可以打出所有的参数数值以及它们在字典中对应的键值来。我们还可以同时计算出这个模型的参数总量为 43。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------\n",
      "KEY: 0.weight\n",
      "shape: (4, 5)\n",
      "tensor([[-0.00335,  0.23990, -0.36808, -0.32912, -0.17225],\n",
      "        [ 0.11992, -0.00886,  0.35459, -0.03969,  0.11834],\n",
      "        [-0.13515, -0.08791, -0.42724, -0.29618, -0.18435],\n",
      "        [ 0.01657,  0.17680,  0.26834, -0.30318, -0.19474]])\n",
      "------\n",
      "KEY: 0.bias\n",
      "shape: (4,)\n",
      "tensor([ 0.16244,  0.37136, -0.09204,  0.33466])\n",
      "------\n",
      "KEY: 2.weight\n",
      "shape: (3, 4)\n",
      "tensor([[-0.08059,  0.05291,  0.45274, -0.46384],\n",
      "        [-0.31477, -0.12658, -0.19490,  0.43200],\n",
      "        [-0.32409, -0.23017, -0.34932, -0.46828]])\n",
      "------\n",
      "KEY: 2.bias\n",
      "shape: (3,)\n",
      "tensor([-0.29187,  0.42980,  0.22311])\n",
      "------\n",
      "KEY: 4.weight\n",
      "shape: (1, 3)\n",
      "tensor([[ 0.27983,  0.03036, -0.29600]])\n",
      "------\n",
      "KEY: 4.bias\n",
      "shape: (1,)\n",
      "tensor([0.09768])\n",
      "======\n",
      "Parameter number: 43\n"
     ]
    }
   ],
   "source": [
    "parameter_number = 0\n",
    "\n",
    "for name, tensor in param.items():\n",
    "    print(\"------\")\n",
    "    print(\"KEY:\", name)\n",
    "    print(\"shape:\", tuple(tensor.shape))\n",
    "    print(tensor)\n",
    "    parameter_number += torch.prod(torch.tensor(tensor.shape))\n",
    "    \n",
    "print(\"======\")\n",
    "print(\"Parameter number:\", int(parameter_number))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们希望取出其中第一个矩阵 `0.weight`，我们可以用下述代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.00335,  0.23990, -0.36808, -0.32912, -0.17225],\n",
       "        [ 0.11992, -0.00886,  0.35459, -0.03969,  0.11834],\n",
       "        [-0.13515, -0.08791, -0.42724, -0.29618, -0.18435],\n",
       "        [ 0.01657,  0.17680,  0.26834, -0.30318, -0.19474]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param[\"0.weight\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `model.state_dict()` 是比较标准的取出参数表的方法；我们之后为了能对参数进行导数计算，这里换用 `model.named_parameters()` 取出参数，并使用其它变量名替代其中的参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "param = {}\n",
    "for i in model.named_parameters():\n",
    "    param[i[0]] = i[1]\n",
    "W_0, b_0 = param[\"0.weight\"], param[\"0.bias\"]\n",
    "W_2, b_2 = param[\"2.weight\"], param[\"2.bias\"]\n",
    "W_4, b_4 = param[\"4.weight\"], param[\"4.bias\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们要解决如下的几个问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一个问题是，键值的名称是如何被定义的。我们现在先回头看一下模型 `model` 是如何定义的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=5, out_features=4, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=4, out_features=3, bias=True)\n",
       "  (3): ReLU()\n",
       "  (4): Linear(in_features=3, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这就意味着，在 PyTorch 中，程序认定这个模型当前是 5 层 (而不是我们刚才提到的 MLP 的 2 层神经元的层数)；其中第 0, 2, 4 层是线性层，1, 3 层是 ReLU 激活层。一般来说，一层神经元是指一层线性层叠加一层激活层；因此我们会说第一层神经元对应 `model` 的 0, 1 层，而第二层神经元对应 `model` 的第 2, 3 层。之后我们通常在提到“层”的时候，指的是程序中出现的 5 层。\n",
    "\n",
    "其中具有参数的层只有线性层。`W_0` 表示的是第 0 层的权重 $\\mathbf{W}^0$，而 `b_0` 表示第 0 层的偏置 $\\mathbf{b}^0$。第 2, 4 层类推。留意在这篇文档中，上标数字并不代表矩阵或向量的幂运算。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二个问题是，每个参数值的维度是如何被定义的。\n",
    "\n",
    "我们仍然需要回头看一下 `model` 的定义。以第 0 层为例，其输入特征长度 `in_features` 为 5，这与 `X` $\\mathbf{X}$ 的维度是相同；而其输出特征长度 `out_features` 为 4。另一个例子是最后一层即第 4 层，其输出特征长度 `out_features` 为 1，这与 `y` $\\mathbf{y}$ 的维度是相同。\n",
    "\n",
    "对于任何线性层而言，其权重矩阵的维度为 (`out_features`, `in_features`)；其偏置向量的维度为 (`out_features`, )。以第 0 层为例，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 5])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W_0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_0.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于权重矩阵与偏置向量，它们的作用将会马上在下一段介绍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 线性层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们简单地了解一下第 0 层线性层的作用。实际上，线性层的作用是相当简单的：\n",
    "\n",
    "$$\n",
    "\\mathbf{X}^0 = \\mathbf{X} \\mathbf{W}^{0,\\mathrm{T}} + \\mathbf{b}^0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的记号中，`X_0` $\\mathbf{X}^0$ 表示第 0 层输出矩阵，`X` $\\mathbf{X}$ 表示第 0 层输入矩阵 (恰好为数据集)，`W_0` $\\mathbf{W}^{0}$ 表示第 0 层作为线性层的权重矩阵，`b_0` $\\mathbf{b}^0$ 表示第 0 层作为线性层的偏置向量。上标 $\\mathrm{T}$ 表示矩阵转置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.02335,  0.18398, -0.18986, -0.25360],\n",
       "        [ 0.07644,  0.59601, -0.43176,  0.86336],\n",
       "        [-1.05575,  1.21783, -0.89241,  0.23785],\n",
       "        [ 1.43460,  0.03912,  0.24424,  0.79060],\n",
       "        [ 0.94343,  0.15788,  0.72290,  0.59383],\n",
       "        [-0.18687,  0.44553, -0.41596,  0.37610]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_0 = X @ param[\"0.weight\"].T + param[\"0.bias\"]\n",
    "X_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但上述矩阵的表示相对来说不太直观；一种相对来说直观的方式可以是 (但不是标准的表示方法，只是作者写多了化学反应式自然会生成的一种习惯)\n",
    "\n",
    "$$\n",
    "\\mathbf{X} \\xrightarrow{\\text{Linear} (5, 4)} \\mathbf{X}^0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于第 2, 4 层线性层，其过程是非常类似的："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{X}^1 &\\xrightarrow{\\text{Linear} (4, 3)} \\mathbf{X}^2 \\\\\n",
    "\\mathbf{X}^3 &\\xrightarrow{\\text{Linear} (3, 1)} \\mathbf{X}^4 = \\mathbf{y}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 激活层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLP 的线性层一般就是普通的矩阵乘法，但激活层的选取可以较为丰富。一种典型与常用的激活层是 ReLU (Rectified Linear Unit)，其定义非常简单：\n",
    "\n",
    "$$\n",
    "\\mathrm{ReLU} (x) = \\max(x, 0)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，若我们记 `X_0` $\\mathbf{X}^0$ 表示第 0 层输出矩阵，`X_1` $\\mathbf{X}^0$ 表示第 1 层输出矩阵；那么\n",
    "\n",
    "$$\n",
    "\\mathbf{X}^1 = \\mathrm{ReLU} (\\mathbf{X}^0) = \\max(\\mathbf{X}^0, \\mathbf{0})\n",
    "$$\n",
    "\n",
    "若联合线性层，其比较直观的表示方式为\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{X} &\\xrightarrow[\\mathrm{ReLU}]{\\text{Linear} (5, 4)} \\mathbf{X}^1 \\\\\n",
    "\\mathbf{X}^1 &\\xrightarrow[\\mathrm{ReLU}]{\\text{Linear} (4, 3)} \\mathbf{X}^3\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于程序的实现上，`X_1` 可以表示为"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.00000, 0.18398, 0.00000, 0.00000],\n",
       "        [0.07644, 0.59601, 0.00000, 0.86336],\n",
       "        [0.00000, 1.21783, 0.00000, 0.23785],\n",
       "        [1.43460, 0.03912, 0.24424, 0.79060],\n",
       "        [0.94343, 0.15788, 0.72290, 0.59383],\n",
       "        [0.00000, 0.44553, 0.00000, 0.37610]], grad_fn=<MaxBackward2>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_1 = torch.max(X_0, torch.tensor(0.))\n",
    "X_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以写出 `model` 模型下，从 $\\mathbf{X}$ 到 $\\mathbf{y}$ 的流程：\n",
    "\n",
    "$$\n",
    "\\mathbf{X}\n",
    "\\xrightarrow[\\mathrm{ReLU}]{\\text{Linear} (5, 4)} \\mathbf{X}^1\n",
    "\\xrightarrow[\\mathrm{ReLU}]{\\text{Linear} (4, 3)} \\mathbf{X}^3\n",
    "\\xrightarrow{\\text{Linear} (3, 1)} \\mathbf{X}^4 = \\mathbf{y}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若要写出每一步的详细过程，则可以是\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{X}^0 &= \\mathbf{X} \\mathbf{W}^{0,\\mathrm{T}} + \\mathbf{b}^0 \\\\\n",
    "\\mathbf{X}^1 &= \\max (\\mathbf{X}^0, \\mathbf{0}) \\\\\n",
    "\\mathbf{X}^2 &= \\mathbf{X}^1 \\mathbf{W}^{2,\\mathrm{T}} + \\mathbf{b}^2 \\\\\n",
    "\\mathbf{X}^3 &= \\max (\\mathbf{X}^2, \\mathbf{0}) \\\\\n",
    "\\mathbf{y} = \\mathbf{X}^4 &= \\mathbf{X}^3 \\mathbf{W}^{4,\\mathrm{T}} + \\mathbf{b}^4\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将上述矩阵的计算操作结果记作 `X_4` $\\mathbf{X}^4$，而仍然将 `model(X)` 的结果记作 `y` $\\mathbf{y}$。我们可以发现这两者是完全等价的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.05652],\n",
       "        [0.11903],\n",
       "        [0.10917],\n",
       "        [0.10579],\n",
       "        [0.10462],\n",
       "        [0.11395]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_0 = X   @ param[\"0.weight\"].T + param[\"0.bias\"]\n",
    "X_1 = torch.max(X_0, torch.tensor(0.))\n",
    "X_2 = X_1 @ param[\"2.weight\"].T + param[\"2.bias\"]\n",
    "X_3 = torch.max(X_2, torch.tensor(0.))\n",
    "X_4 = X_3 @ param[\"4.weight\"].T + param[\"4.bias\"]\n",
    "X_4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.05652],\n",
       "        [0.11903],\n",
       "        [0.10917],\n",
       "        [0.10579],\n",
       "        [0.10462],\n",
       "        [0.11395]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们希望在一个公式内解决一切与 $\\mathbf{X}$ 到 $\\mathbf{y}$ 的映射，我们可以写作\n",
    "\n",
    "$$\n",
    "\\mathbf{X} \\xrightarrow{\\text{MLP}} \\mathbf{y} \\Leftrightarrow\n",
    "\\mathbf{y} = \\mathbf{X}^4 = \\max \\left[ \\max (\\mathbf{X} \\mathbf{W}^{0,\\mathrm{T}} + \\mathbf{b}^0, \\mathbf{0}) \\mathbf{W}^{2,\\mathrm{T}} + \\mathbf{b}^2, \\mathbf{0} \\right] \\mathbf{W}^{4,\\mathrm{T}} + \\mathbf{b}^4\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.05652],\n",
       "        [0.11903],\n",
       "        [0.10917],\n",
       "        [0.10579],\n",
       "        [0.10462],\n",
       "        [0.11395]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_4 = torch.max(torch.max(\n",
    "    X @ param[\"0.weight\"].T + param[\"0.bias\"],\n",
    "    torch.tensor(0.)) @ param[\"2.weight\"].T + param[\"2.bias\"],\n",
    "    torch.tensor(0.)) @ param[\"4.weight\"].T + param[\"4.bias\"]\n",
    "X_4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此，我们已经成功地用简单的矩阵运算还原了 PyTorch 封装过的 MLP 模型 `model` 了。这是可以是一个比较典型的模型重复的标准途径，也将会对文档第三部分了解模型的反向传播过程有重要的意义。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLP 参数导数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 被求导对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们就以这个非常短小、但是需要使用链式法则进行反向传播的 MLP 模型 `model` 进行分析。我们的目标是与 PyTorch 的 Autograd 的结果进行核对。但我们在上一份文档中提到，任何函数的导数都必须转化为标量的导数；即使输出并非是标量，也必须乘以与输出相同维度的张量并作求和得到变量。\n",
    "\n",
    "对于当前的问题而言，作为标量的值是输出 `y` $\\mathbf{y}$ 与真实值 `t` $\\mathbf{t}$ 的 L1 损失函数值 `loss` $\\mathrm{loss}$：\n",
    "\n",
    "$$\n",
    "\\mathrm{loss} = \\Vert \\mathbf{y} - \\mathbf{t} \\Vert_1\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.96400, grad_fn=<L1LossBackward>)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = nn.L1Loss()(y, t)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这一段的分析都是基于 `loss` 的导数得到。对该值的导数可以通过下述方式得到："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grad of W_0\n",
      "tensor([[ 0.00151,  0.00503, -0.00132, -0.00306, -0.00254],\n",
      "        [-0.00285,  0.00299,  0.00823, -0.01116, -0.00508],\n",
      "        [ 0.00160,  0.00226, -0.00186, -0.00189, -0.00106],\n",
      "        [ 0.00024, -0.00386, -0.00327,  0.00378, -0.00053]])\n",
      "grad of b_0\n",
      "tensor([ 0.00478, -0.00751,  0.00197, -0.01093])\n",
      "grad of W_2\n",
      "tensor([[ 0.00000,  0.00000,  0.00000,  0.00000],\n",
      "        [-0.01242, -0.01336, -0.00489, -0.01448],\n",
      "        [ 0.00000,  0.00908,  0.00000,  0.00000]])\n",
      "grad of b_2\n",
      "tensor([ 0.00000, -0.03036,  0.04933])\n",
      "grad of W_4\n",
      "tensor([[ 0.00000, -0.41996, -0.03013]])\n",
      "grad of b_4\n",
      "tensor([-1.])\n"
     ]
    }
   ],
   "source": [
    "model.zero_grad()\n",
    "loss.backward()\n",
    "print(\"grad of W_0\"); print(W_0.grad)\n",
    "print(\"grad of b_0\"); print(b_0.grad)\n",
    "print(\"grad of W_2\"); print(W_2.grad)\n",
    "print(\"grad of b_2\"); print(b_2.grad)\n",
    "print(\"grad of W_4\"); print(W_4.grad)\n",
    "print(\"grad of b_4\"); print(b_4.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第 4 层参数导数：非参数矩阵引入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在所有的参数中，最容易求的导数在最后一层。我们将 `loss` $\\mathrm{loss}$ 写为第 2 层参数的函数：\n",
    "\n",
    "$$\n",
    "\\mathrm{loss} = \\Vert \\mathbf{X}^3 \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\Vert_1\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.96400, grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(X_3 @ W_4.T + b_4 - t).abs().sum() / t.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但从公式的表达角度上，上面的表达式不太容易被求导。在此，我们引入第 5 层非参数矩阵 `l_5` $\\mathbf{l}^5 = \\mathrm{dim} (\\mathbf{y})^{-1} \\mathrm{sgn} (\\mathbf{y} - \\mathbf{t})$，那么\n",
    "\n",
    "$$\n",
    "\\mathrm{loss} = \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathbf{X}^3 \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\right)\n",
    "$$\n",
    "\n",
    "或表示为矩阵元的形式，\n",
    "\n",
    "$$\n",
    "\\mathrm{loss} = \\sum_{ij} l_{i0}^{5} \\left( X_{ij}^3 W_{0j}^{4} + b_0^4 - t_{i0} \\right)\n",
    "$$\n",
    "\n",
    "上式中出现的 $0$ 表示该维度恰好为 1 维。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 1])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_5 = (y - t).sign() / y.shape[0]\n",
    "l_5.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.96400]], grad_fn=<MmBackward>)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_5.T @ (X_3 @ W_4.T + b_4 - t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据上面公式的表述，我们就可以比较容易地给出关于 $\\mathbf{W}^{4, \\mathrm{T}}$ 的导数\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial \\mathbf{W}^{4}} = \\mathbf{l}^{5, \\mathrm{T}} \\mathbf{X}^3\n",
    "$$\n",
    "\n",
    "或矩阵元的形式\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial W_{0j}^{4}} = \\sum_{i} l_{i0}^{5} X_{ij}^3\n",
    "$$\n",
    "\n",
    "我们可以将其 `gW_4` 与 PyTorch 自动导数给出的 `W_4.grad` 进行比对："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gW_4 = l_5.T @ X_3\n",
    "torch.allclose(gW_4, W_4.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但这里需要指出，由于 $\\mathbf{b}^4$ 作为偏置矩阵的维度与其它矩阵不太相同，因此在求梯度时，需要对 $\\mathbf{l}^{5, \\mathrm{T}}$ 的第一维度，或者说对表示数据点数量的维度求和：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial b_0^4} = \\sum_{i} l_{i0}^{5}\n",
    "$$\n",
    "\n",
    "我们可以将其 `gb_4` 与 PyTorch 自动导数给出的 `b_4.grad` 进行比对："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb_4 = l_5.T.sum(dim=-1)\n",
    "torch.allclose(gb_4, b_4.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第 2 层参数导数：链式法则"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二层参数的导数会复杂不少。如果我们把 `loss` $\\mathrm{loss}$ 写为第 2 层参数的函数："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\mathrm{loss} = \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathrm{ReLU} \\left( \\mathbf{X}^1 \\mathbf{W}^{2, \\mathrm{T}} + \\mathbf{b}^2 \\right) \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.96400]], grad_fn=<MmBackward>)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_5.T @ (nn.ReLU()(X_1 @ W_2.T + b_2) @ W_4.T + b_4 - t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果要尝试对 $\\mathbf{W}^{2}$ 或 $\\mathbf{b}^2$ 求导，我们又会遇到与刚才第四层导数相同的问题，即如何对 $\\mathrm{ReLU}$ 求导。我们引入第 3 层非参数矩阵 `l_3` $\\mathbf{l}^3 = \\mathrm{sgn}^+ (\\mathbf{X}^{2})$：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathrm{loss}\n",
    "&= \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathbf{X}^3 \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\right) \\\\\n",
    "&= \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathbf{l}^3 \\odot \\left( \\mathbf{X}^1 \\mathbf{W}^{2, \\mathrm{T}} + \\mathbf{b}^2 \\right) \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "或表示为矩阵元的形式，\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathrm{loss}\n",
    "&= \\sum_{ij} l_{i0}^{5} \\left( X_{ij}^3 W_{0j}^{4} + b_0^4 - t_{i0} \\right) \\\\\n",
    "&= \\sum_{ijk} l_{i0}^{5} \\left( l_{ij}^3 (X_{ik}^1 W_{jk}^2 + b_{j}^2) W_{0j}^{4} + b_0^4 - t_{i0} \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "其中，记号 $\\mathrm{sgn}^+$ 表示正值取 1，负值取 0：\n",
    "\n",
    "$$\n",
    "\\mathrm{sgn}^+ (x) = \\max( \\mathrm{sgn} (x), 0 )\n",
    "$$\n",
    "\n",
    "上面的计算过程中，矩阵 elementwise 乘法与普通的矩阵乘法的运算优先级是相同的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgn_pos = lambda t: torch.max(t.sign(), torch.tensor(0.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 3])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_3 = sgn_pos(X_2)\n",
    "l_3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.96400]], grad_fn=<MmBackward>)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_5.T @ (l_3 * (X_1 @ W_2.T + b_2) @ W_4.T + b_4 - t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们固然可以对上式直接求关于 $\\mathbf{W}^{2}$ 参数的导数得到 `gW_2`，并与 PyTorch 自动求导给出的 `W_2.grad` 进行比对：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial \\mathbf{W}^{2}} = \\mathbf{W}^{4, \\mathrm{T}} \\mathbf{l}^{5, \\mathrm{T}} \\odot \\mathbf{l}^{3, \\mathrm{T}} \\mathbf{X}^1\n",
    "$$\n",
    "\n",
    "或\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial W_{jk}^2} = \\sum_{i} l_{ij}^3 (X_{ik}^1 W_{jk}^2 + b_{j}^2) W_{0j}^{4}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gW_2 = W_4.T @ l_5.T * l_3.T @ X_1\n",
    "torch.allclose(gW_2, W_2.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但我们可以有更为精妙的做法 (或者说思考方法)。注意到我们可以将 $\\mathrm{loss}$ 看成关于 $\\mathbf{X}^3$ 的函数，而 $\\mathbf{X}^3$ 又可以看作是关于 $\\mathbf{W}^2$ 的函数：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{X}^3 &= \\mathbf{l}^3 \\odot \\left( \\mathbf{X}^1 \\mathbf{W}^{2, \\mathrm{T}} + \\mathbf{b}^2 \\right) \\\\\n",
    "\\mathrm{loss} &= \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathbf{X}^3 \\mathbf{W}^{4, \\mathrm{T}} + \\mathbf{b}^4 - \\mathbf{t} \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "或\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "X_{ij}^3 &= \\sum_{k} l_{ij}^3 (X_{ik}^1 W_{jk}^2 + b_{j}^2) \\\\\n",
    "\\mathrm{loss} &= \\sum_{ij} l_{i0}^{5} \\left( X_{ij}^3 W_{0j}^{4} + b_0^4 - t_{i0} \\right) \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "那么，我们就可以分别写出导数\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3} &= l_{i0}^{5} W_{0j}^{4} \\\\\n",
    "\\frac{\\partial X_{ij}^3}{\\partial W_{jk}^2} &= l_{ij}^3 X_{ik}^1\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "我们定义 $\\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3}$ 在程序中表示为 `gX_3`。依据链式法则，我们就可以给出\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial W_{jk}^2} = \\sum_i \\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3} \\frac{\\partial X_{ij}^3}{\\partial W_{jk}^2} = \\sum_i \\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3} l_{ij}^3 X_{ik}^1\n",
    "$$\n",
    "\n",
    "拿上式与 PyTorch 自动求导给出的 `W_2.grad` 进行比对也能得到正确的结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gX_3 = l_5 @ W_4\n",
    "gW_2 = gX_3.T * l_3.T @ X_1\n",
    "torch.allclose(gW_2, W_2.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而对于 $\\mathbf{b}^2$ 的导数 `gb_2` 的情况也会非常类似：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial b_{j}^2} = \\sum_i \\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3} \\frac{\\partial X_{ij}^3}{\\partial b_{j}^2} = \\sum_i \\frac{\\partial \\mathrm{loss}}{\\partial X_{ij}^3} l_{ij}^3\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb_2 = (gX_3.T * l_3.T).sum(dim=-1)\n",
    "torch.allclose(gb_2, b_2.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也许在这个例子中没有感受到链式法则的意义；即使不使用链式法则，程序的复杂性也没有增大许多。但对于 $\\mathbf{W}^0$ 与 $\\mathbf{b}^0$，使用链式法则的意义就会表现出来。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第 0 层参数导数：反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面一小段中，在求导过程中我们使用了一次链式法则。下面我们会比较系统地，从头回顾，将问题一步一步拆分，从而获得第 0 层参数 $\\mathbf{W}^0$ 与 $\\mathbf{b}^0$ 的导数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{X}^0 &= \\mathbf{X} \\mathbf{W}^{0,\\mathrm{T}} + \\mathbf{b}^0 \\tag{0} \\\\\n",
    "\\mathbf{X}^1 &= \\mathbf{l}^1 \\odot \\mathbf{X}^0 \\tag{1} \\\\\n",
    "\\mathbf{X}^2 &= \\mathbf{X}^1 \\mathbf{W}^{2,\\mathrm{T}} + \\mathbf{b}^2 \\tag{2} \\\\\n",
    "\\mathbf{X}^3 &= \\mathbf{l}^3 \\odot \\mathbf{X}^2 \\tag{3} \\\\\n",
    "\\mathbf{X}^4 &= \\mathbf{X}^3 \\mathbf{W}^{4,\\mathrm{T}} + \\mathbf{b}^4 \\tag{4} \\\\\n",
    "\\mathrm{loss} &= \\mathbf{l}^{5, \\mathrm{T}} \\left( \\mathbf{X}^4 - \\mathbf{t} \\right) \\tag{5}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述的表示在写程序时可能是方便的，但在分析时，容易会对维度的信息把握得不太好。我们在此将下标明确地写出：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "X^0_{ik} &= \\sum_l X_{il} W^0_{kl} + b^0_{k} \\tag{0} \\\\\n",
    "X^1_{ik} &= l^1_{ik} X^0_{ik} \\tag{1} \\\\\n",
    "X^2_{ij} &= \\sum_k X^1_{ik} W^2_{jk} + b^2_{j} \\tag{2} \\\\\n",
    "X^3_{ij} &= l^3_{ij} X^2_{ij} \\tag{3} \\\\\n",
    "X^4_{i0} &= \\sum_j X^3_{ij} W^4_{0j} + b^4_{0} \\tag{4} \\\\\n",
    "\\mathrm{loss} &= \\sum_i l^5_{i0} (X^4_{i0} - t_{i0}) \\tag{5}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中的第 1, 3, 5 层引入了非参数矩阵，它们的定义为\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "l^1_{ik} &= \\mathrm{sgn}^+ (X^0_{ik}) \\\\\n",
    "l^3_{ij} &= \\mathrm{sgn}^+ (X^2_{ij}) \\\\\n",
    "l^5_{i0} &= \\mathrm{dim} (i)^{-1} \\mathrm{sgn} (X^4_{i0} - t_{i0})\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "l_1 = sgn_pos(X_0)\n",
    "l_3 = sgn_pos(X_2)\n",
    "l_5 = y.shape[0]**-1 * (y - t).sign()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的目标是第 0 层参数导数 $\\mathbf{W}^0$ 与 $\\mathbf{b}^0$，但我们的被求导对象 $\\mathrm{loss}$ 在第 5 层。反向传播的一种解读方式是，导数的求取是从最后层向前传播。那么我们一层一层地对导数进行求取。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 5 层** `gX_4` $\\frac{\\partial \\mathrm{loss}}{\\partial X^4_{i0}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X^4_{i0}} = l^5_{i0}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "gX_4 = l_5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 4 层** `gX_3` $\\frac{\\partial \\mathrm{loss}}{\\partial X^3_{ij}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^4_{i0}}{\\partial X^3_{ij}} = W^4_{0j}, \\quad\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X^3_{ij}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^4_{i0}} \\frac{\\partial X^4_{i0}}{\\partial X^3_{ij}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^4_{i0}} W^4_{0j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "gX_3 = gX_4 @ W_4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 3 层** `gX_2` $\\frac{\\partial \\mathrm{loss}}{\\partial X^2_{ij}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^3_{ij}}{\\partial X^2_{ij}} = l^3_{ij}, \\quad\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X^2_{ij}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^3_{ij}} \\frac{\\partial X^3_{ij}}{\\partial X^2_{ij}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^3_{ij}} l^3_{ij}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "gX_2 = gX_3 * l_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 2 层** `gX_1` $\\frac{\\partial \\mathrm{loss}}{\\partial X^1_{ik}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^2_{ij}}{\\partial X^1_{ik}} = W^2_{jk}, \\quad\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X^1_{ik}} = \\sum_j \\frac{\\partial \\mathrm{loss}}{\\partial X^2_{ij}} \\frac{\\partial X^2_{ij}}{\\partial X^1_{ik}} = \\sum_j \\frac{\\partial \\mathrm{loss}}{\\partial X^2_{ij}} W^2_{jk}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "gX_1 = gX_2 @ W_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 1 层** `gX_0` $\\frac{\\partial \\mathrm{loss}}{\\partial X^0_{ik}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^1_{ik}}{\\partial X^0_{ik}} = l^1_{ik}, \\quad\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial X^0_{ik}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^1_{ik}} \\frac{\\partial X^1_{ik}}{\\partial X^0_{ik}} = \\frac{\\partial \\mathrm{loss}}{\\partial X^1_{ik}} l^1_{ik}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "gX_0 = gX_1 * l_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**第 0 层**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`gW_0` $\\frac{\\partial \\mathrm{loss}}{\\partial W^0_{kl}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^0_{ik}}{\\partial W^0_{kl}} = X_{il}, \\quad\n",
    "\\frac{\\mathrm{loss}}{\\partial W^0_{kl}} = \\sum_i \\frac{\\mathrm{loss}}{\\partial X^0_{ik}} \\frac{\\partial X^0_{ik}}{\\partial W^0_{kl}} = \\sum_i \\frac{\\mathrm{loss}}{\\partial X^0_{ik}} X_{il}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gW_0 = gX_0.T @ X\n",
    "torch.allclose(gW_0, W_0.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`gb_0` $\\frac{\\partial \\mathrm{loss}}{\\partial b^0_{k}}$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial X^0_{ik}}{\\partial b^0_{k}} = 1, \\quad\n",
    "\\frac{\\mathrm{loss}}{\\partial b^0_{k}} = \\sum_i \\frac{\\mathrm{loss}}{\\partial X^0_{ik}} \\frac{\\partial X^0_{ik}}{\\partial b^0_{k}} = \\sum_i \\frac{\\mathrm{loss}}{\\partial X^0_{ik}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb_0 = gX_0.sum(dim=0)\n",
    "torch.allclose(gb_0, b_0.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我们已经成功地使用了反向传播的方法，对最难求解的 $\\mathbf{W}^0$ 与 $\\mathbf{b}^0$ 的参数进行了导数求取。这样一个过程可以是非常清晰与规范化的。\n",
    "\n",
    "对于该模型更简单的其它参数，或者对于层数更多的 MLP 网络，使用完全相同的方式也能生成所有的导数。\n",
    "\n",
    "我们知道，深度网络的参数优化过程通常是梯度下降的衍生方法。如果知道了这些参数的导数量，我们至少相信可以很自然地写出一个 naive 梯度下降优参的程序，实现深度网络的学习。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
